Systems and methods for detecting traffic signal details

ABSTRACT

Systems and methods use cameras to provide autonomous navigation features. In one implementation, a traffic light detection system is disclosed. The system may include at least one image capture device configured to acquire a plurality of images of an area forward of the vehicle, the area including a traffic lamp fixture having at least one traffic light. At least one processing device may be configured to align areas of the plurality of images corresponding to the traffic light, based on a determined center of brightness, expand each pixel within the aligned areas, determine a set of average pixel values including an average pixel value for each set of corresponding expanded pixels within the aligned areas, and determine, based on the set of average pixel values, whether the traffic light includes an arrow.

CROSS REFERENCES TO RELATED APPLICATIONS

This application claims the benefit of priority of U.S. ProvisionalPatent Application No. 61/933,325, filed on Jan. 30, 2014; U.S.Provisional Patent Application No. 61/993,111, filed on May 14, 2014;U.S. Provisional Patent Application No. 62/040,246, filed on Aug. 21,2014; U.S. Provisional Patent Application No. 62/060,603, filed on Oct.7, 2014; and U.S. Provisional Patent Application No. 62/102,669, filedon Jan. 13, 2015. All of the foregoing applications are incorporatedherein by reference in their entirety.

BACKGROUND

I. Technical Field

The present disclosure relates generally to autonomous vehiclenavigation and, more specifically, to systems and methods that usecameras to provide autonomous vehicle navigation features.

II. Background Information

As technology continues to advance, the goal of a fully autonomousvehicle that is capable of navigating on roadways is on the horizon.Primarily, an autonomous vehicle may be able to identify its environmentand navigate without input from a human operator. Autonomous vehiclesmay also take into account a variety of factors and make appropriatedecisions based on those factors to safely and accurately reach anintended destination. For example, various objects—such as othervehicles and pedestrians—are encountered when a vehicle typicallytravels a roadway. Autonomous driving systems may recognize theseobjects in a vehicle's environment and take appropriate and timelyaction to avoid collisions. Additionally, autonomous driving systems mayidentify other indicators—such as traffic signals, traffic signs, andlane markings—that regulate vehicle movement (e.g., when the vehiclemust stop and may go, a speed at which the vehicle must not exceed,where the vehicle must be positioned on the roadway, etc.). Autonomousdriving systems may need to determine when a vehicle should changelanes, turn at intersections, change roadways, etc. As is evident fromthese examples, many factors may need to be addressed in order toprovide an autonomous vehicle that is capable of navigating safely andaccurately.

SUMMARY

Embodiments consistent with the present disclosure provide systems andmethods for autonomous vehicle navigation. The disclosed embodiments mayuse cameras to provide autonomous vehicle navigation features. Forexample, consistent with the disclosed embodiments, the disclosedsystems may include one, two, or more cameras that monitor theenvironment of a vehicle and cause a navigational response based on ananalysis of images captured by one or more of the cameras.

Consistent with a disclosed embodiment, a lane ending detection systemis provided for a vehicle. The system may include at least one imagecapture device configured to acquire at least one image of an areaforward of the vehicle, the area including at least one road signproviding information indicative of a lane ending; a data interface; andat least one processing device. The at least one processing device maybe configured to: receive the at least one image via the data interface;extract lane ending information from a representation of the at leastone road sign included in the at least one image; determine, based on atleast one indicator of position of the vehicle, a distance from thevehicle to one or more lane constraints associated with the current lanein which the vehicle is traveling; determine, based on the extractedlane ending information and based on the determined distance from thevehicle to the one or more lane constraints, whether a current lane inwhich the vehicle is traveling is ending; and cause the vehicle tochange lanes based on the determination of whether the current lane inwhich the vehicle is traveling is ending.

Consistent with another disclosed embodiment, a vehicle may include abody; at least one image capture device configured to acquire at leastone image of an area forward of the vehicle, the area including at leastone road sign providing information indicative of a lane ending; a datainterface; and at least one processing device. The at least oneprocessing device may be configured to: receive the at least one imagevia the data interface; extract lane ending information from arepresentation of the at least one road sign included in the at leastone image; determine, based on at least one indicator of position of thevehicle, a distance from the vehicle to one or more lane constraintsassociated with the current lane in which the vehicle is traveling;determine, based on the extracted lane information and based on thedetermined distance from the vehicle to the one or more laneconstraints, whether a current lane in which the vehicle is traveling isending; and cause the vehicle to change lanes based on the determinationof whether the current lane in which the vehicle is traveling is ending.

Consistent with another disclosed embodiment, a method is provided fordetecting a lane ending for a vehicle. The method may include acquiring,via at least one image capture device, at least one image of an areaforward of the vehicle, the area including at least one road signproviding information indicative of the lane ending; extracting laneending information from a representation of the at least one road signincluded in the at least one image; determining, based on at least oneindicator of position of the vehicle, a distance from the vehicle to oneor more lane constraints associated with the current lane in which thevehicle is traveling; determining, based on the extracted lane endinginformation and based on the determined distance from the vehicle to theone or more lane constraints, whether a current lane in which thevehicle is traveling is ending; and causing the vehicle to change lanesbased on the determination of whether the current lane in which thevehicle is traveling is ending.

Consistent with a disclosed embodiment, a driver-assist object detectionsystem is provided for a vehicle. The system may include at least oneimage capture device configured to capture a plurality of imagesrepresentative of an area surrounding the vehicle; a data interface; andat least one processing device. The at least one processing device maybe configured to: receive, via the data interface, a first image fromamong the plurality of images and at least a second image from among theplurality of images; analyze the first image and at least the secondimage to determine a reference plane corresponding to a road plane;locate a target object in the first image and in the second image;determine a difference in a size of at least one dimension of the targetobject between the first image and the second image; use the determineddifference in the size of the at least one dimension to determine atleast a height of the target object; and cause a change in at least adirectional course of the vehicle if the determined height of the objectexceeds a predetermined threshold.

Consistent with another disclosed embodiment, a vehicle may include abody; at least one image capture device configured to capture aplurality of images representative of an area surrounding the vehicle; adata interface; and at least one processing device. The at least oneprocessing device may be configured to: receive, via the data interface,a first image from among the plurality of images and at least a secondimage from among the plurality of images; analyze the first image and atleast the second image to determine a reference place corresponding to aroad plane; locate a target object in the first image and in the secondimage; determine a difference in a size of at least one dimension of thetarget object between the first image and the second image; use thedetermined difference in the size of the at least one dimension todetermine at least a height of the target object; and cause a change inat least a directional course of the vehicle if the determined height ofthe object exceeds a predetermined threshold.

Consistent with another disclosed embodiment, a method is provided fordetecting an object in a roadway. The method may include capturing, viaat least one image capture device, a plurality of images representativeof an area surrounding the vehicle; receiving a first image from amongthe plurality of images and at least a second image from among theplurality of images; analyzing the first image and at least the secondimage to determine a reference plane corresponding to a road plane;determining a difference in a size of at least one dimension of thetarget object between the first image and the second image; using thedetermined difference in the size of the at least one dimension todetermine at least a height of the target object; and causing a changein at least a directional course of the vehicle if the determined heightof the object exceeds a predetermined threshold.

Consistent with a disclosed embodiment, a traffic light detection systemis provided for a vehicle. The system may include at least one imagecapture device configured to acquire a plurality of images of a trafficlight located in a region above the vehicle, outside of a sightline of atypical driver location in the vehicle, a data interface; and at leastone processing device. The at least one processing device may beconfigured to: receive the plurality of images via the data interface;determine a status of the traffic light based on analysis of at least afirst image from among the plurality of images; determine a change instatus of the traffic light based on at least a second image from amongthe plurality of images; and cause a system response based on thedetermination of the change in status.

Consistent with another disclosed embodiment, a vehicle may include abody; at least one image capture device configured to acquire aplurality of images of a traffic light located in a region above thevehicle, outside of a sightline of a typical driver location in vehicle;a data interface; and at least one processing device. The at least oneprocessing device may be configured to: receive the plurality of imagesvia the data interface; determine a status of the traffic light based onanalysis of at least a first image from among the plurality of images;determine a change in status of the traffic light based on at least asecond image from among the plurality of images; and cause a systemresponse based on the determination of the change in status.

Consistent with another disclosed embodiment, a method is provided fornavigating a vehicle. The method may include: acquiring, via at leastone image capture device, a plurality of images of a traffic lightlocated in a region above the vehicle, outside of a sightline of atypical driver location in vehicle; determining a status of the trafficlight based on analysis of at least a first image from among theplurality of images; determining a change in status of the traffic lightbased on at least a second image from among the plurality of images; andcausing a system response based on the determination of the change instatus.

Consistent with a disclosed embodiment, a traffic light detection systemis provided for a vehicle. The system may include at least one imagecapture device configured to acquire at least one image of an areaforward of the vehicle, the area including a plurality of traffic lampfixtures each including at least one traffic light. They system mayfurther include a data interface and at least one processing device. Theat least one processing device may be configured to: receive the atleast one acquired image via the data interface; use at least oneindicator of vehicle position, as determined from the at least oneacquired image, to determine a relevance to the vehicle of each of theplurality of traffic lamp fixtures; determine, based on the at least oneacquired image, a status of a traffic light included in at least onetraffic lamp determined to be relevant to the vehicle; and cause asystem response based on the determined status.

Consistent with another disclosed embodiment, a vehicle may include abody; at least one image capture device configured to acquire at leastone image of an area forward of the vehicle, the area including aplurality of traffic lamp fixtures each including at least one trafficlight; a data interface; and at least one processing device. The atleast one processing device may be configured to: receive the at leastone acquired image via the data interface; use at least one indicator ofvehicle position, as determined from the at least one acquired image, todetermine a relevance to the vehicle of each of the plurality of trafficlamp fixtures; determine, based on the at least one acquired image, astatus of a traffic light included in at least one traffic lamp fixturedetermined to be relevant to the vehicle; and cause a system responsebased on the determined status.

Consistent with another disclosed embodiment, a method is provided fortraffic light detection. The method may include acquiring, via at leastone image capture device, at least one image of an area forward of avehicle, the area including a plurality of traffic lamp fixtures eachincluding at least one traffic light; using at least one indicator ofvehicle position, as determined from the at least one acquired image, todetermine a relevance to the vehicle of each of the plurality of trafficlamp fixtures; determining, based on the at least one acquired image, astatus of a traffic light included in at least one traffic lamp fixturedetermined to be relevant to the vehicle; and causing a system responsebased on the determined status.

Consistent with a disclosed embodiment, a traffic light detection systemis provided for a vehicle. The system may include at least one imagecapture device configured to acquire at least one image of an areaforward of the vehicle, the area including a traffic lamp fixture havingat least one traffic light; a data interface; and at least oneprocessing device. The at least one processing device may be configuredto: receive, via the data interface, the at least one acquired image;determine, based on at least one indicator of vehicle position, whetherthe vehicle is in a turn lane; perform image processing on the at leastone image to determine whether an arrow exists in the at least one imagewithin pixels of the image representative of the traffic light;determine, based on the at least one image, a status of the trafficlight; and cause a system response based on the determination of thestatus of the traffic light, whether the traffic light includes anarrow, and whether the vehicle is in a turn lane.

Consistent with another disclosed embodiment, a vehicle may include abody; at least one image capture device configured to acquire at leastone image of an area forward of the vehicle, the area including atraffic lamp fixture having at least one traffic light; a datainterface; and at least one processing device. The at least oneprocessing device may be configured to: receive, via the data interface,the at least one acquired image; determine, based on at least oneindicator of vehicle position, whether the vehicle is in a turn lane;perform image processing on the at least one image to determine whetheran arrow exists in the at least one image within pixels of the imagerepresentative of the traffic light; determine, based on the at leastone image, a status of the traffic light; and cause a system responsebased on the determination of the status of the traffic light, whetherthe traffic light includes an arrow, and whether the vehicle is in aturn lane.

Consistent with another disclosed embodiment, a method is provided fortraffic light detection. The method may include acquiring, via at leastone image capture device, at least one image of an area forward of avehicle, the area including a traffic lamp fixture having at least onetraffic light; determining, based on at least one indicator of vehicleposition, whether the vehicle is in a turn lane; performing imageprocessing on the at least one image to determine whether an arrowexists in the at least one image within pixels of the imagerepresentative of the traffic light; determining, based on the at leastone image, a status of the traffic light; and causing a system responsebased on the determination of the status of the traffic light, whetherthe traffic light includes an arrow, and whether the vehicle is in aturn lane.

Consistent with a disclosed embodiment, a traffic light detection systemis provided. The system may include at least one image capture deviceconfigured to acquire a plurality of images of an area forward of thevehicle, the area including a traffic lamp fixture having at least onetraffic light, a data interface, and at least one processing deviceconfigured to receive the plurality of images via the data interface.The at least one processing device may be further configured to alignareas of the plurality of images corresponding to the traffic light,based on a determined center of brightness, expand each pixel within thealigned areas, determine a set of average pixel values including anaverage pixel value for each set of corresponding expanded pixels withinthe aligned areas, and determine, based on the set of average pixelvalues, whether the traffic light includes an arrow.

Consistent with another disclosed embodiment, a vehicle is provided. Thevehicle may include a body, at least one image capture device configuredto acquire a plurality of images of an area forward of the vehicle, thearea including a traffic lamp fixture having at least one traffic light,a data interface, and at least one processing device configured toreceive the plurality of images via the data interface. The at least oneprocessing device may be further configured to align areas of theplurality of images corresponding to the traffic light, based on adetermined center of brightness, expand each pixel within the alignedareas, determine a set of average pixel values including an averagepixel value for each set of corresponding expanded pixels within thealigned areas, and determine, based on the set of average pixel values,whether the traffic light includes an arrow.

Consistent with another disclosed embodiment, a method of traffic lightdetection is provided. The method may include acquiring, via at leastone image capture device, a plurality of images of an area forward of avehicle, the area including a traffic lamp fixture having at least onetraffic light, aligning areas of the plurality of images correspondingto the traffic light, based on a determined center of brightness, andexpanding each pixel within the aligned areas. The method may alsoinclude determining a set of average pixel values including an averagepixel value for each set of corresponding expanded pixels within thealigned areas, and determining, based on the set of average pixelvalues, whether the traffic light includes an arrow.

Consistent with other disclosed embodiments, non-transitorycomputer-readable storage media may store program instructions, whichare executed by at least one processing device and perform any of themethods described herein.

The foregoing general description and the following detailed descriptionare exemplary and explanatory only and are not restrictive of theclaims.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this disclosure, illustrate various disclosed embodiments. Inthe drawings:

FIG. 1 is a diagrammatic representation of an exemplary systemconsistent with the disclosed embodiments.

FIG. 2A is a diagrammatic side view representation of an exemplaryvehicle including a system consistent with the disclosed embodiments.

FIG. 2B is a diagrammatic top view representation of the vehicle andsystem shown in FIG. 2A consistent with the disclosed embodiments.

FIG. 2C is a diagrammatic top view representation of another embodimentof a vehicle including a system consistent with the disclosedembodiments.

FIG. 2D is a diagrammatic top view representation of yet anotherembodiment of a vehicle including a system consistent with the disclosedembodiments.

FIG. 2E is a diagrammatic top view representation of yet anotherembodiment of a vehicle including a system consistent with the disclosedembodiments.

FIG. 2F is a diagrammatic representation of exemplary vehicle controlsystems consistent with the disclosed embodiments.

FIG. 3A is a diagrammatic representation of an interior of a vehicleincluding a rearview mirror and a user interface for a vehicle imagingsystem consistent with the disclosed embodiments.

FIG. 3B is an illustration of an example of a camera mount that isconfigured to be positioned behind a rearview mirror and against avehicle windshield consistent with the disclosed embodiments.

FIG. 3C is an illustration of the camera mount shown in FIG. 3B from adifferent perspective consistent with the disclosed embodiments.

FIG. 3D is an illustration of an example of a camera mount that isconfigured to be positioned behind a rearview mirror and against avehicle windshield consistent with the disclosed embodiments.

FIG. 4 is an exemplary block diagram of a memory configured to storeinstructions for performing one or more operations consistent with thedisclosed embodiments.

FIG. 5A is a flowchart showing an exemplary process for causing one ormore navigational responses based on monocular image analysis consistentwith disclosed embodiments.

FIG. 5B is a flowchart showing an exemplary process for detecting one ormore vehicles and/or pedestrians in a set of images consistent with thedisclosed embodiments.

FIG. 5C is a flowchart showing an exemplary process for detecting roadmarks and/or lane geometry information in a set of images consistentwith the disclosed embodiments.

FIG. 5D is a flowchart showing an exemplary process for detectingtraffic lights in a set of images consistent with the disclosedembodiments.

FIG. 5E is a flowchart showing an exemplary process for causing one ormore navigational responses based on a vehicle path consistent with thedisclosed embodiments.

FIG. 5F is a flowchart showing an exemplary process for determiningwhether a leading vehicle is changing lanes consistent with thedisclosed embodiments.

FIG. 6 is a flowchart showing an exemplary process for causing one ormore navigational responses based on stereo image analysis consistentwith the disclosed embodiments.

FIG. 7 is a flowchart showing an exemplary process for causing one ormore navigational responses based on an analysis of three sets of imagesconsistent with the disclosed embodiments.

FIG. 8 is a diagrammatic top view representation of an exemplary vehicleincluding a system consistent with the disclosed embodiments in whichthe vehicle's current travel lane is ending.

FIG. 9 is a diagrammatic representation of the memory of an exemplarylane end recognition system consistent with the disclosed embodiments.

FIG. 10A is a diagrammatic representation of an exemplary vehicleincluding a lane end recognition system encountering the end of a travellane marked by road signs consistent with the disclosed embodiments.

FIG. 10B is a diagrammatic representation of an exemplary vehicleincluding a lane end recognition system traveling a measurable distancebehind a lead vehicle consistent with the disclosed embodiments.

FIG. 11 is a flowchart showing an exemplary process for determiningwhether a current lane in which the vehicle is traveling is ending,consistent with the disclosed embodiments.

FIG. 12 is a diagrammatic top view representation of an exemplaryvehicle including a system consistent with the disclosed embodiments inwhich an object is detected within the travel lane in which the vehicleis traveling.

FIG. 13 is a diagrammatic representation of the memory of an exemplarydriver-assist object detection system consistent with the disclosedembodiments.

FIG. 14A is a diagrammatic representation of two captured imagesindicating an object found within the travel lane among a plurality ofimages captured by an exemplary driver-assist object detection systemconsistent with the disclosed embodiments.

FIG. 14B is a diagrammatic top view representation of an exemplaryvehicle including a system consistent with the disclosed embodiments inwhich an object is detected within the travel lane in which the vehicleis traveling.

FIG. 15 is a flowchart showing an exemplary process for determiningwhether a current lane in which the vehicle is traveling is ending,consistent with the disclosed embodiments.

FIG. 16 is a diagrammatic side view representation of an exemplaryvehicle including an image capture device with an extended angle-of-viewconsistent with the disclosed embodiments.

FIG. 17 is a diagrammatic top view representation of the vehicleincluding the image capture device with the extended angle-of-viewconsistent with the disclosed embodiments.

FIG. 18 is a diagrammatic representation of a memory storinginstructions to identify objects located outside of a sightline of atypical driver location consistent with the disclosed embodiments.

FIG. 19 is a flowchart showing an exemplary process for identifyingobjects located outside of a sightline of a typical driver locationconsistent with the disclosed embodiments.

FIG. 20 is a diagrammatic representation of two exemplary vehiclesapproaching an intersection with a plurality of traffic lightsconsistent with the disclosed embodiments.

FIG. 21 is a diagrammatic representation of an exemplary vehicleapproaching an intersection with a plurality of traffic lightsconsistent with the disclosed embodiments.

FIG. 22 is a diagrammatic representation of a memory storinginstructions for detecting traffic lights consistent with the disclosedembodiments.

FIG. 23 is a flowchart showing an exemplary process for detectingtraffic lights consistent with the disclosed embodiments.

FIG. 24 is a diagrammatic top view representation of an exemplaryvehicle including a system consistent with the disclosed embodiments inwhich the vehicle is approaching an intersection with a traffic light.

FIG. 25 is a diagrammatic representation of the memory of an exemplarytraffic light detection system consistent with the disclosedembodiments.

FIG. 26A is a diagrammatic representation of an exemplary vehicleincluding a traffic light detection system encountering an intersectionwith a traffic light and determining its status consistent with thedisclosed embodiments.

FIG. 26B is a diagrammatic representation of an exemplary vehicleincluding a traffic light detection system encountering an intersectionwith a blinking yellow light and determining a course of actionconsistent with the disclosed embodiments.

FIG. 27 is a flowchart showing an exemplary process for determining thestatus of a traffic light at an intersection, consistent with thedisclosed embodiments.

FIG. 28 is an exemplary block diagram of a memory configured to storeinstructions for performing one or more operations consistent withdisclosed embodiments.

FIG. 29A is an illustration of an exemplary traffic light, consistentwith disclosed embodiments.

FIG. 29B is another illustration of an exemplary traffic light,consistent with disclosed embodiments.

FIG. 29C is an illustration of an exemplary image of a portion of thetraffic light of FIG. 29A, consistent with disclosed embodiments.

FIG. 29D is an illustration of an exemplary image of a portion of thetraffic light of FIG. 29B, consistent with disclosed embodiments.

FIG. 30A is a flowchart showing an exemplary process for determiningwhether a traffic light includes an arrow, consistent with disclosedembodiments.

FIG. 30B is a flowchart showing an exemplary process for causing asystem response based on the detection of an arrow signal in a trafficlight, consistent with disclosed embodiments.

DETAILED DESCRIPTION

The following detailed description refers to the accompanying drawings.Wherever possible, the same reference numbers are used in the drawingsand the following description to refer to the same or similar parts.While several illustrative embodiments are described herein,modifications, adaptations and other implementations are possible. Forexample, substitutions, additions or modifications may be made to thecomponents illustrated in the drawings, and the illustrative methodsdescribed herein may be modified by substituting, reordering, removing,or adding steps to the disclosed methods. Accordingly, the followingdetailed description is not limited to the disclosed embodiments andexamples. Instead, the proper scope is defined by the appended claims.

Disclosed embodiments provide systems and methods that use cameras toprovide autonomous navigation features. In various embodiments, thesystem may include one, two or more cameras that monitor the environmentof a vehicle. In one embodiment, the system may recognize when a drivinglane will end and, when an end is detected, make a responsivenavigational change, such as a lane change. Rather than relying on mapdata to make this determination, the system may operate based solely onvisual information. Further, while the system may recognize an end of alane by direct observation of the lane ending, the system may recognizean approaching lane end sooner by, for example, observing andrecognizing road sign alerts that provide information regarding a laneending. In another embodiment, the system may determine the presence ofobstacles in a roadway where those obstacles have a height of less than,for example, 10 cm. In another embodiment, the system may view andrecognize traffic lights located above a vehicle. In another embodiment,the system may recognize a traffic light and its status from among aplurality of traffic lights and cause a navigational change based on thestatus of the traffic light (e.g., discontinue cruise and cause brakingwhen a red light is recognized). For example, the system may recognizewhich of the imaged traffic lights are relevant to the vehicle andrespond only to those traffic lights determined to be relevant. Inanother embodiment, the system may recognize a turn lane traffic lightand its status based on analysis of road context information combinedwith determined characteristics of the light. In another embodiment, thesystem may employ a super resolution technique to recognize and analyzetraffic lights. For example, the system may distinguish between roundand arrow shaped traffic lights.

FIG. 1 is a block diagram representation of a system 100 consistent withthe exemplary disclosed embodiments. System 100 may include variouscomponents depending on the requirements of a particular implementation.In some embodiments, system 100 may include a processing unit 110, animage acquisition unit 120, a position sensor 130, one or more memoryunits 140, 150, a map database 160, and a user interface 170. Processingunit 110 may include one or more processing devices. In someembodiments, processing unit 110 may include an applications processor180, an image processor 190, or any other suitable processing device.Similarly, image acquisition unit 120 may include any number of imageacquisition devices and components depending on the requirements of aparticular application. In some embodiments, image acquisition unit 120may include one or more image capture devices (e.g., cameras), such asimage capture device 122, image capture device 124, and image capturedevice 126. System 100 may also include a data interface 128communicatively connecting processing device 110 to image acquisitiondevice 120. For example, data interface 128 may include any wired and/orwireless link or links for transmitting image data acquired by imageaccusation device 120 to processing unit 110.

Both applications processor 180 and image processor 190 may includevarious types of processing devices. For example, either or both ofapplications processor 180 and image processor 190 may include amicroprocessor, preprocessors (such as an image preprocessor), graphicsprocessors, a central processing unit (CPU), support circuits, digitalsignal processors, integrated circuits, memory, or any other types ofdevices suitable for running applications and for image processing andanalysis. In some embodiments, applications processor 180 and/or imageprocessor 190 may include any type of single or multi-core processor,mobile device microcontroller, central processing unit, etc. Variousprocessing devices may be used, including, for example, processorsavailable from manufacturers such as Intel®, AMD®, etc. and may includevarious architectures (e.g., x86 processor, ARM®, etc.).

In some embodiments, applications processor 180 and/or image processor190 may include any of the EyeQ series of processor chips available fromMobileye®. These processor designs each include multiple processingunits with local memory and instruction sets. Such processors mayinclude video inputs for receiving image data from multiple imagesensors and may also include video out capabilities. In one example, theEyeQ2® uses 90 nm-micron technology operating at 332 Mhz. The EyeQ2®architecture consists of two floating point, hyper-thread 32-bit RISCCPUs (MIPS32® 34K® cores), five Vision Computing Engines (VCE), threeVector Microcode Processors (VMP®), Denali 64-bit Mobile DDR Controller,128-bit internal Sonics Interconnect, dual 16-bit Video input and 18-bitVideo output controllers, 16 channels DMA and several peripherals. TheMIPS34K CPU manages the five VCEs, three VMP™ and the DMA, the secondMIPS34K CPU and the multi-channel DMA as well as the other peripherals.The five VCEs, three VMP® and the MIPS34K CPU can perform intensivevision computations required by multi-function bundle applications. Inanother example, the EyeQ3®, which is a third generation processor andis six times more powerful that the EyeQ2®, may be used in the disclosedembodiments.

Any of the processing devices disclosed herein may be configured toperform certain functions. Configuring a processing device, such as anyof the described EyeQ processors or other controller or microprocessor,to perform certain functions may include programming of computerexecutable instructions and making those instructions available to theprocessing device for execution during operation of the processingdevice. In some embodiments, configuring a processing device may includeprogramming the processing device directly with architecturalinstructions. In other embodiments, configuring a processing device mayinclude storing executable instructions on a memory that is accessibleto the processing device during operation. For example, the processingdevice may access the memory to obtain and execute the storedinstructions during operation.

While FIG. 1 depicts two separate processing devices included inprocessing unit 110, more or fewer processing devices may be used. Forexample, in some embodiments, a single processing device may be used toaccomplish the tasks of applications processor 180 and image processor190. In other embodiments, these tasks may be performed by more than twoprocessing devices.

Processing unit 110 may comprise various types of devices. For example,processing unit 110 may include various devices, such as a controller,an image preprocessor, a central processing unit (CPU), supportcircuits, digital signal processors, integrated circuits, memory, or anyother types of devices for image processing and analysis. The imagepreprocessor may include a video processor for capturing, digitizing andprocessing the imagery from the image sensors. The CPU may comprise anynumber of microcontrollers or microprocessors. The support circuits maybe any number of circuits generally well known in the art, includingcache, power supply, clock and input-output circuits. The memory maystore software that, when executed by the processor, controls theoperation of the system. The memory may include databases and imageprocessing software. The memory may comprise any number of random accessmemories, read only memories, flash memories, disk drives, opticalstorage, tape storage, removable storage and other types of storage. Inone instance, the memory may be separate from the processing unit 110.In another instance, the memory may be integrated into the processingunit 110.

Each memory 140, 150 may include software instructions that whenexecuted by a processor (e.g., applications processor 180 and/or imageprocessor 190), may control operation of various aspects of system 100.These memory units may include various databases and image processingsoftware. The memory units may include random access memory, read onlymemory, flash memory, disk drives, optical storage, tape storage,removable storage and/or any other types of storage. In someembodiments, memory units 140, 150 may be separate from the applicationsprocessor 180 and/or image processor 190. In other embodiments, thesememory units may be integrated into applications processor 180 and/orimage processor 190.

Position sensor 130 may include any type of device suitable fordetermining a location associated with at least one component of system100. In some embodiments, position sensor 130 may include a GPSreceiver. Such receivers can determine a user position and velocity byprocessing signals broadcasted by global positioning system satellites.Position information from position sensor 130 may be made available toapplications processor 180 and/or image processor 190.

User interface 170 may include any device suitable for providinginformation to or for receiving inputs from one or more users of system100. In some embodiments, user interface 170 may include user inputdevices, including, for example, a touchscreen, microphone, keyboard,pointer devices, track wheels, cameras, knobs, buttons, etc. With suchinput devices, a user may be able to provide information inputs orcommands to system 100 by typing instructions or information, providingvoice commands, selecting menu options on a screen using buttons,pointers, or eye-tracking capabilities, or through any other suitabletechniques for communicating information to system 100.

User interface 170 may be equipped with one or more processing devicesconfigured to provide and receive information to or from a user andprocess that information for use by, for example, applications processor180. In some embodiments, such processing devices may executeinstructions for recognizing and tracking eye movements, receiving andinterpreting voice commands, recognizing and interpreting touches and/orgestures made on a touchscreen, responding to keyboard entries or menuselections, etc. In some embodiments, user interface 170 may include adisplay, speaker, tactile device, and/or any other devices for providingoutput information to a user.

Map database 160 may include any type of database for storing map datauseful to system 100. In some embodiments, map database 160 may includedata relating to the position, in a reference coordinate system, ofvarious items, including roads, water features, geographic features,businesses, points of interest, restaurants, gas stations, etc. Mapdatabase 160 may store not only the locations of such items, but alsodescriptors relating to those items, including, for example, namesassociated with any of the stored features. In some embodiments, mapdatabase 160 may be physically located with other components of system100. Alternatively or additionally, map database 160 or a portionthereof may be located remotely with respect to other components ofsystem 100 (e.g., processing unit 110). In such embodiments, informationfrom map database 160 may be downloaded over a wired or wireless dataconnection to a network (e.g., over a cellular network and/or theInternet, etc.).

Image capture devices 122, 124, and 126 may each include any type ofdevice suitable for capturing at least one image from an environment.Moreover, any number of image capture devices may be used to acquireimages for input to the image processor. Some embodiments may includeonly a single image capture device, while other embodiments may includetwo, three, or even four or more image capture devices. Image capturedevices 122, 124, and 126 will be further described with reference toFIGS. 2B-2E, below.

System 100, or various components thereof, may be incorporated intovarious different platforms. In some embodiments, system 100 may beincluded on a vehicle 200, as shown in FIG. 2A. For example, vehicle 200may be equipped with a processing unit 110 and any of the othercomponents of system 100, as described above relative to FIG. 1. Whilein some embodiments vehicle 200 may be equipped with only a single imagecapture device (e.g., camera), in other embodiments, such as thosediscussed in connection with FIGS. 2B-2E, multiple image capture devicesmay be used. For example, either of image capture devices 122 and 124 ofvehicle 200, as shown in FIG. 2A, may be part of an ADAS (AdvancedDriver Assistance Systems) imaging set.

The image capture devices included on vehicle 200 as part of the imageacquisition unit 120 may be positioned at any suitable location. In someembodiments, as shown in FIGS. 2A-2E and 3A-3C, image capture device 122may be located in the vicinity of the rearview mirror. This position mayprovide a line of sight similar to that of the driver of vehicle 200,which may aid in determining what is and is not visible to the driver.Image capture device 122 may be positioned at any location near therearview mirror, but placing image capture device 122 on the driver sideof the mirror may further aid in obtaining images representative of thedriver's field of view and/or line of sight.

Other locations for the image capture devices of image acquisition unit120 may also be used. For example, image capture device 124 may belocated on or in a bumper of vehicle 200. Such a location may beespecially suitable for image capture devices having a wide field ofview. The line of sight of bumper-located image capture devices can bedifferent from that of the driver and, therefore, the bumper imagecapture device and driver may not always see the same objects. The imagecapture devices (e.g., image capture devices 122, 124, and 126) may alsobe located in other locations. For example, the image capture devicesmay be located on or in one or both of the side mirrors of vehicle 200,on the roof of vehicle 200, on the hood of vehicle 200, on the trunk ofvehicle 200, on the sides of vehicle 200, mounted on, positioned behind,or positioned in front of any of the windows of vehicle 200, and mountedin or near light figures on the front and/or back of vehicle 200, etc.

In addition to image capture devices, vehicle 200 may include variousother components of system 100. For example, processing unit 110 may beincluded on vehicle 200 either integrated with or separate from anengine control unit (ECU) of the vehicle. Vehicle 200 may also beequipped with a position sensor 130, such as a GPS receiver and may alsoinclude a map database 160 and memory units 140 and 150.

FIG. 2A is a diagrammatic side view representation of an exemplaryvehicle imaging system consistent with the disclosed embodiments. FIG.2B is a diagrammatic top view illustration of the embodiment shown inFIG. 2A. As illustrated in FIG. 2B, the disclosed embodiments mayinclude a vehicle 200 including in its body a system 100 with a firstimage capture device 122 positioned in the vicinity of the rearviewmirror and/or near the driver of vehicle 200, a second image capturedevice 124 positioned on or in a bumper region (e.g., one of bumperregions 210) of vehicle 200, and a processing unit 110.

As illustrated in FIG. 2C, image capture devices 122 and 124 may both bepositioned in the vicinity of the rearview mirror and/or near the driverof vehicle 200. Additionally, while two image capture devices 122 and124 are shown in FIGS. 2B and 2C, it should be understood that otherembodiments may include more than two image capture devices. Forexample, in the embodiments shown in FIGS. 2D and 2E, first, second, andthird image capture devices 122, 124, and 126, are included in thesystem 100 of vehicle 200.

As illustrated in FIG. 2D, image capture device 122 may be positioned inthe vicinity of the rearview mirror and/or near the driver of vehicle200, and image capture devices 124 and 126 may be positioned on or in abumper region (e.g., one of bumper regions 210) of vehicle 200. And asshown in FIG. 2E, image capture devices 122, 124, and 126 may bepositioned in the vicinity of the rearview mirror and/or near the driverseat of vehicle 200. The disclosed embodiments are not limited to anyparticular number and configuration of the image capture devices, andthe image capture devices may be positioned in any appropriate locationwithin and/or on vehicle 200.

It is to be understood that the disclosed embodiments are not limited tovehicles and could be applied in other contexts. It is also to beunderstood that disclosed embodiments are not limited to a particulartype of vehicle 200 and may be applicable to all types of vehiclesincluding automobiles, trucks, trailers, and other types of vehicles.

The first image capture device 122 may include any suitable type ofimage capture device. Image capture device 122 may include an opticalaxis. In one instance, the image capture device 122 may include anAptina M9V024 WVGA sensor with a global shutter. In other embodiments,image capture device 122 may provide a resolution of 1280×960 pixels andmay include a rolling shutter. Image capture device 122 may includevarious optical elements. In some embodiments one or more lenses may beincluded, for example, to provide a desired focal length and field ofview for the image capture device. In some embodiments, image capturedevice 122 may be associated with a 6 mm lens or a 12 mm lens. In someembodiments, image capture device 122 may be configured to captureimages having a desired field-of-view (FOV) 202, as illustrated in FIG.2D. For example, image capture device 122 may be configured to have aregular FOV, such as within a range of 40 degrees to 56 degrees,including a 46 degree FOV, 50 degree FOV, 52 degree FOV, or greater.Alternatively, image capture device 122 may be configured to have anarrow FOV in the range of 23 to 40 degrees, such as a 28 degree FOV or36 degree FOV. In addition, image capture device 122 may be configuredto have a wide FOV in the range of 100 to 180 degrees. In someembodiments, image capture device 122 may include a wide angle bumpercamera or one with up to a 180 degree FOV.

The first image capture device 122 may acquire a plurality of firstimages relative to a scene associated with the vehicle 200. Each of theplurality of first images may be acquired as a series of image scanlines, which may be captured using a rolling shutter. Each scan line mayinclude a plurality of pixels.

The first image capture device 122 may have a scan rate associated withacquisition of each of the first series of image scan lines. The scanrate may refer to a rate at which an image sensor can acquire image dataassociated with each pixel included in a particular scan line.

Image capture devices 122, 124, and 126 may contain any suitable typeand number of image sensors, including CCD sensors or CMOS sensors, forexample. In one embodiment, a CMOS image sensor may be employed alongwith a rolling shutter, such that each pixel in a row is read one at atime, and scanning of the rows proceeds on a row-by-row basis until anentire image frame has been captured. In some embodiments, the rows maybe captured sequentially from top to bottom relative to the frame.

The use of a rolling shutter may result in pixels in different rowsbeing exposed and captured at different times, which may cause skew andother image artifacts in the captured image frame. On the other hand,when the image capture device 122 is configured to operate with a globalor synchronous shutter, all of the pixels may be exposed for the sameamount of time and during a common exposure period. As a result, theimage data in a frame collected from a system employing a global shutterrepresents a snapshot of the entire FOV (such as FOV 202) at aparticular time. In contrast, in a rolling shutter application, each rowin a frame is exposed and data is capture at different times. Thus,moving objects may appear distorted in an image capture device having arolling shutter. This phenomenon will be described in greater detailbelow.

The second image capture device 124 and the third image capturing device126 may be any type of image capture device. Like the first imagecapture device 122, each of image capture devices 124 and 126 mayinclude an optical axis. In one embodiment, each of image capturedevices 124 and 126 may include an Aptina M9V024 WVGA sensor with aglobal shutter. Alternatively, each of image capture devices 124 and 126may include a rolling shutter. Like image capture device 122, imagecapture devices 124 and 126 may be configured to include various lensesand optical elements. In some embodiments, lenses associated with imagecapture devices 124 and 126 may provide FOVs (such as FOVs 204 and 206)that are the same as, or narrower than, a FOV (such as FOV 202)associated with image capture device 122. For example, image capturedevices 124 and 126 may have FOVs of 40 degrees, 30 degrees, 26 degrees,23 degrees, 20 degrees, or less.

Image capture devices 124 and 126 may acquire a plurality of second andthird images relative to a scene associated with the vehicle 200. Eachof the plurality of second and third images may be acquired as a secondand third series of image scan lines, which may be captured using arolling shutter. Each scan line or row may have a plurality of pixels.Image capture devices 124 and 126 may have second and third scan ratesassociated with acquisition of each of image scan lines included in thesecond and third series.

Each image capture device 122, 124, and 126 may be positioned at anysuitable position and orientation relative to vehicle 200. The relativepositioning of the image capture devices 122, 124, and 126 may beselected to aid in fusing together the information acquired from theimage capture devices. For example, in some embodiments, a FOV (such asFOV 204) associated with image capture device 124 may overlap partiallyor fully with a FOV (such as FOV 202) associated with image capturedevice 122 and a FOV (such as FOV 206) associated with image capturedevice 126.

Image capture devices 122, 124, and 126 may be located on vehicle 200 atany suitable relative heights. In one instance, there may be a heightdifference between the image capture devices 122, 124, and 126, whichmay provide sufficient parallax information to enable stereo analysis.For example, as shown in FIG. 2A, the two image capture devices 122 and124 are at different heights. There may also be a lateral displacementdifference between image capture devices 122, 124, and 126, givingadditional parallax information for stereo analysis by processing unit110, for example. The difference in the lateral displacement may bedenoted by d_(x), as shown in FIGS. 2C and 2D. In some embodiments, foreor aft displacement (e.g., range displacement) may exist between imagecapture devices 122, 124, and 126. For example, image capture device 122may be located 0.5 to 2 meters or more behind image capture device 124and/or image capture device 126. This type of displacement may enableone of the image capture devices to cover potential blind spots of theother image capture device(s).

Image capture devices 122 may have any suitable resolution capability(e.g., number of pixels associated with the image sensor), and theresolution of the image sensor(s) associated with the image capturedevice 122 may be higher, lower, or the same as the resolution of theimage sensor(s) associated with image capture devices 124 and 126. Insome embodiments, the image sensor(s) associated with image capturedevice 122 and/or image capture devices 124 and 126 may have aresolution of 640×480, 1024×768, 1280×960, or any other suitableresolution.

The frame rate (e.g., the rate at which an image capture device acquiresa set of pixel data of one image frame before moving on to capture pixeldata associated with the next image frame) may be controllable. Theframe rate associated with image capture device 122 may be higher,lower, or the same as the frame rate associated with image capturedevices 124 and 126. The frame rate associated with image capturedevices 122, 124, and 126 may depend on a variety of factors that mayaffect the timing of the frame rate. For example, one or more of imagecapture devices 122, 124, and 126 may include a selectable pixel delayperiod imposed before or after acquisition of image data associated withone or more pixels of an image sensor in image capture device 122, 124,and/or 126. Generally, image data corresponding to each pixel may beacquired according to a clock rate for the device (e.g., one pixel perclock cycle). Additionally, in embodiments including a rolling shutter,one or more of image capture devices 122, 124, and 126 may include aselectable horizontal blanking period imposed before or afteracquisition of image data associated with a row of pixels of an imagesensor in image capture device 122, 124, and/or 126. Further, one ormore of image capture devices 122, 124, and/or 126 may include aselectable vertical blanking period imposed before or after acquisitionof image data associated with an image frame of image capture device122, 124, and 126.

These timing controls may enable synchronization of frame ratesassociated with image capture devices 122, 124, and 126, even where theline scan rates of each are different. Additionally, as will bediscussed in greater detail below, these selectable timing controls,among other factors (e.g., image sensor resolution, maximum line scanrates, etc.) may enable synchronization of image capture from an areawhere the FOV of image capture device 122 overlaps with one or more FOVsof image capture devices 124 and 126, even where the field of view ofimage capture device 122 is different from the FOVs of image capturedevices 124 and 126.

Frame rate timing in image capture device 122, 124, and 126 may dependon the resolution of the associated image sensors. For example, assumingsimilar line scan rates for both devices, if one device includes animage sensor having a resolution of 640×480 and another device includesan image sensor with a resolution of 1280×960, then more time will berequired to acquire a frame of image data from the sensor having thehigher resolution.

Another factor that may affect the timing of image data acquisition inimage capture devices 122, 124, and 126 is the maximum line scan rate.For example, acquisition of a row of image data from an image sensorincluded in image capture device 122, 124, and 126 will require someminimum amount of time. Assuming no pixel delay periods are added, thisminimum amount of time for acquisition of a row of image data will berelated to the maximum line scan rate for a particular device. Devicesthat offer higher maximum line scan rates have the potential to providehigher frame rates than devices with lower maximum line scan rates. Insome embodiments, one or more of image capture devices 124 and 126 mayhave a maximum line scan rate that is higher than a maximum line scanrate associated with image capture device 122. In some embodiments, themaximum line scan rate of image capture device 124 and/or 126 may be1.25, 1.5, 1.75, or 2 times or more than a maximum line scan rate ofimage capture device 122.

In another embodiment, image capture devices 122, 124, and 126 may havethe same maximum line scan rate, but image capture device 122 may beoperated at a scan rate less than or equal to its maximum scan rate. Thesystem may be configured such that one or more of image capture devices124 and 126 operate at a line scan rate that is equal to the line scanrate of image capture device 122. In other instances, the system may beconfigured such that the line scan rate of image capture device 124and/or image capture device 126 may be 1.25, 1.5, 1.75, or 2 times ormore than the line scan rate of image capture device 122.

In some embodiments, image capture devices 122, 124, and 126 may beasymmetric. That is, they may include cameras having different fields ofview (FOV) and focal lengths. The fields of view of image capturedevices 122, 124, and 126 may include any desired area relative to anenvironment of vehicle 200, for example. In some embodiments, one ormore of image capture devices 122, 124, and 126 may be configured toacquire image data from an environment in front of vehicle 200, behindvehicle 200, to the sides of vehicle 200, or combinations thereof.

Further, the focal length associated with each image capture device 122,124, and/or 126 may be selectable (e.g., by inclusion of appropriatelenses etc.) such that each device acquires images of objects at adesired distance range relative to vehicle 200. For example, in someembodiments image capture devices 122, 124, and 126 may acquire imagesof close-up objects within a few meters from the vehicle. Image capturedevices 122, 124, and 126 may also be configured to acquire images ofobjects at ranges more distant from the vehicle (e.g., 25 m, 50 m, 100m, 150 m, or more). Further, the focal lengths of image capture devices122, 124, and 126 may be selected such that one image capture device(e.g., image capture device 122) can acquire images of objectsrelatively close to the vehicle (e.g., within 10 m or within 20 m) whilethe other image capture devices (e.g., image capture devices 124 and126) can acquire images of more distant objects (e.g., greater than 20m, 50 m, 100 m, 150 m, etc.) from vehicle 200.

According to some embodiments, the FOV of one or more image capturedevices 122, 124, and 126 may have a wide angle. For example, it may beadvantageous to have a FOV of 140 degrees, especially for image capturedevices 122, 124, and 126 that may be used to capture images of the areain the vicinity of vehicle 200. For example, image capture device 122may be used to capture images of the area to the right or left ofvehicle 200 and, in such embodiments, it may be desirable for imagecapture device 122 to have a wide FOV (e.g., at least 140 degrees).

The field of view associated with each of image capture devices 122,124, and 126 may depend on the respective focal lengths. For example, asthe focal length increases, the corresponding field of view decreases.

Image capture devices 122, 124, and 126 may be configured to have anysuitable fields of view. In one particular example, image capture device122 may have a horizontal FOV of 46 degrees, image capture device 124may have a horizontal FOV of 23 degrees, and image capture device 126may have a horizontal FOV in between 23 and 46 degrees. In anotherinstance, image capture device 122 may have a horizontal FOV of 52degrees, image capture device 124 may have a horizontal FOV of 26degrees, and image capture device 126 may have a horizontal FOV inbetween 26 and 52 degrees. In some embodiments, a ratio of the FOV ofimage capture device 122 to the FOVs of image capture device 124 and/orimage capture device 126 may vary from 1.5 to 2.0. In other embodiments,this ratio may vary between 1.25 and 2.25.

System 100 may be configured so that a field of view of image capturedevice 122 overlaps, at least partially or fully, with a field of viewof image capture device 124 and/or image capture device 126. In someembodiments, system 100 may be configured such that the fields of viewof image capture devices 124 and 126, for example, fall within (e.g.,are narrower than) and share a common center with the field of view ofimage capture device 122. In other embodiments, the image capturedevices 122, 124, and 126 may capture adjacent FOVs or may have partialoverlap in their FOVs. In some embodiments, the fields of view of imagecapture devices 122, 124, and 126 may be aligned such that a center ofthe narrower FOV image capture devices 124 and/or 126 may be located ina lower half of the field of view of the wider FOV device 122.

FIG. 2F is a diagrammatic representation of exemplary vehicle controlsystems, consistent with the disclosed embodiments. As indicated in FIG.2F, vehicle 200 may include throttling system 220, braking system 230,and steering system 240. System 100 may provide inputs (e.g., controlsignals) to one or more of throttling system 220, braking system 230,and steering system 240 over one or more data links (e.g., any wiredand/or wireless link or links for transmitting data). For example, basedon analysis of images acquired by image capture devices 122, 124, and/or126, system 100 may provide control signals to one or more of throttlingsystem 220, braking system 230, and steering system 240 to navigatevehicle 200 (e.g., by causing an acceleration, a turn, a lane shift,etc.). Further, system 100 may receive inputs from one or more ofthrottling system 220, braking system 230, and steering system 24indicating operating conditions of vehicle 200 (e.g., speed, whethervehicle 200 is braking and/or turning, etc.). Further details areprovided in connection with FIGS. 4-7, below.

As shown in FIG. 3A, vehicle 200 may also include a user interface 170for interacting with a driver or a passenger of vehicle 200. Forexample, user interface 170 in a vehicle application may include a touchscreen 320, knobs 330, buttons 340, and a microphone 350. A driver orpassenger of vehicle 200 may also use handles (e.g., located on or nearthe steering column of vehicle 200 including, for example, turn signalhandles), buttons (e.g., located on the steering wheel of vehicle 200),and the like, to interact with system 100. In some embodiments,microphone 350 may be positioned adjacent to a rearview mirror 310.Similarly, in some embodiments, image capture device 122 may be locatednear rearview mirror 310. In some embodiments, user interface 170 mayalso include one or more speakers 360 (e.g., speakers of a vehicle audiosystem). For example, system 100 may provide various notifications(e.g., alerts) via speakers 360.

FIGS. 3B-3D are illustrations of an exemplary camera mount 370configured to be positioned behind a rearview mirror (e.g., rearviewmirror 310) and against a vehicle windshield, consistent with disclosedembodiments. As shown in FIG. 3B, camera mount 370 may include imagecapture devices 122, 124, and 126. Image capture devices 124 and 126 maybe positioned behind a glare shield 380, which may be flush against thevehicle windshield and include a composition of film and/oranti-reflective materials. For example, glare shield 380 may bepositioned such that it aligns against a vehicle windshield having amatching slope. In some embodiments, each of image capture devices 122,124, and 126 may be positioned behind glare shield 380, as depicted, forexample, in FIG. 3D. The disclosed embodiments are not limited to anyparticular configuration of image capture devices 122, 124, and 126,camera mount 370, and glare shield 380. FIG. 3C is an illustration ofcamera mount 370 shown in FIG. 3B from a front perspective.

As will be appreciated by a person skilled in the art having the benefitof this disclosure, numerous variations and/or modifications may be madeto the foregoing disclosed embodiments. For example, not all componentsare essential for the operation of system 100. Further, any componentmay be located in any appropriate part of system 100 and the componentsmay be rearranged into a variety of configurations while providing thefunctionality of the disclosed embodiments. Therefore, the foregoingconfigurations are examples and, regardless of the configurationsdiscussed above, system 100 can provide a wide range of functionality toanalyze the surroundings of vehicle 200 and navigate vehicle 200 inresponse to the analysis.

As discussed below in further detail and consistent with variousdisclosed embodiments, system 100 may provide a variety of featuresrelated to autonomous driving and/or driver assist technology. Forexample, system 100 may analyze image data, position data (e.g., GPSlocation information), map data, speed data, and/or data from sensorsincluded in vehicle 200. System 100 may collect the data for analysisfrom, for example, image acquisition unit 120, position sensor 130, andother sensors. Further, system 100 may analyze the collected data todetermine whether or not vehicle 200 should take a certain action, andthen automatically take the determined action without humanintervention. For example, when vehicle 200 navigates without humanintervention, system 100 may automatically control the braking,acceleration, and/or steering of vehicle 200 (e.g., by sending controlsignals to one or more of throttling system 220, braking system 230, andsteering system 240). Further, system 100 may analyze the collected dataand issue warnings and/or alerts to vehicle occupants based on theanalysis of the collected data. Additional details regarding the variousembodiments that are provided by system 100 are provided below.

Forward-Facing Multi-Imaging System

As discussed above, system 100 may provide drive assist functionalitythat uses a multi-camera system. The multi-camera system may use one ormore cameras facing in the forward direction of a vehicle. In otherembodiments, the multi-camera system may include one or more camerasfacing to the side of a vehicle or to the rear of the vehicle. In oneembodiment, for example, system 100 may use a two-camera imaging system,where a first camera and a second camera (e.g., image capture devices122 and 124) may be positioned at the front and/or the sides of avehicle (e.g., vehicle 200). The first camera may have a field of viewthat is greater than, less than, or partially overlapping with, thefield of view of the second camera. In addition, the first camera may beconnected to a first image processor to perform monocular image analysisof images provided by the first camera, and the second camera may beconnected to a second image processor to perform monocular imageanalysis of images provided by the second camera. The outputs (e.g.,processed information) of the first and second image processors may becombined. In some embodiments, the second image processor may receiveimages from both the first camera and second camera to perform stereoanalysis. In another embodiment, system 100 may use a three-cameraimaging system where each of the cameras has a different field of view.Such a system may, therefore, make decisions based on informationderived from objects located at varying distances both forward and tothe sides of the vehicle. References to monocular image analysis mayrefer to instances where image analysis is performed based on imagescaptured from a single point of view (e.g., from a single camera).Stereo image analysis may refer to instances where image analysis isperformed based on two or more images captured with one or morevariations of an image capture parameter. For example, captured imagessuitable for performing stereo image analysis may include imagescaptured: from two or more different positions, from different fields ofview, using different focal lengths, along with parallax information,etc.

For example, in one embodiment, system 100 may implement a three cameraconfiguration using image capture devices 122-126. In such aconfiguration, image capture device 122 may provide a narrow field ofview (e.g., 34 degrees, or other values selected from a range of about20 to 45 degrees, etc.), image capture device 124 may provide a widefield of view (e.g., 150 degrees or other values selected from a rangeof about 100 to about 180 degrees), and image capture device 126 mayprovide an intermediate field of view (e.g., 46 degrees or other valuesselected from a range of about 35 to about 60 degrees). In someembodiments, image capture device 126 may act as a main or primarycamera. Image capture devices 122-126 may be positioned behind rearviewmirror 310 and positioned substantially side-by-side (e.g., 6 cm apart).Further, in some embodiments, as discussed above, one or more of imagecapture devices 122-126 may be mounted behind glare shield 380 that isflush with the windshield of vehicle 200. Such shielding may act tominimize the impact of any reflections from inside the car on imagecapture devices 122-126.

In another embodiment, as discussed above in connection with FIGS. 3Band 3C, the wide field of view camera (e.g., image capture device 124 inthe above example) may be mounted lower than the narrow and main fieldof view cameras (e.g., image devices 122 and 126 in the above example).This configuration may provide a free line of sight from the wide fieldof view camera. To reduce reflections, the cameras may be mounted closeto the windshield of vehicle 200, and may include polarizers on thecameras to damp reflected light.

A three camera system may provide certain performance characteristics.For example, some embodiments may include an ability to validate thedetection of objects by one camera based on detection results fromanother camera. In the three camera configuration discussed above,processing unit 110 may include, for example, three processing devices(e.g., three EyeQ series of processor chips, as discussed above), witheach processing device dedicated to processing images captured by one ormore of image capture devices 122-126.

In a three camera system, a first processing device may receive imagesfrom both the main camera and the narrow field of view camera, andperform vision processing of the narrow FOV camera to, for example,detect other vehicles, pedestrians, lane marks, traffic signs, trafficlights, and other road objects. Further, the first processing device maycalculate a disparity of pixels between the images from the main cameraand the narrow camera and create a 3D reconstruction of the environmentof vehicle 200. The first processing device may then combine the 3Dreconstruction with 3D map data or with 3D information calculated basedon information from another camera.

The second processing device may receive images from main camera andperform vision processing to detect other vehicles, pedestrians, lanemarks, traffic signs, traffic lights, and other road objects.Additionally, the second processing device may calculate a cameradisplacement and, based on the displacement, calculate a disparity ofpixels between successive images and create a 3D reconstruction of thescene (e.g., a structure from motion). The second processing device maysend the structure from motion based 3D reconstruction to the firstprocessing device to be combined with the stereo 3D images.

The third processing device may receive images from the wide FOV cameraand process the images to detect vehicles, pedestrians, lane marks,traffic signs, traffic lights, and other road objects. The thirdprocessing device may further execute additional processing instructionsto analyze images to identify objects moving in the image, such asvehicles changing lanes, pedestrians, etc.

In some embodiments, having streams of image-based information capturedand processed independently may provide an opportunity for providingredundancy in the system. Such redundancy may include, for example,using a first image capture device and the images processed from thatdevice to validate and/or supplement information obtained by capturingand processing image information from at least a second image capturedevice.

In some embodiments, system 100 may use two image capture devices (e.g.,image capture devices 122 and 124) in providing navigation assistancefor vehicle 200 and use a third image capture device (e.g., imagecapture device 126) to provide redundancy and validate the analysis ofdata received from the other two image capture devices. For example, insuch a configuration, image capture devices 122 and 124 may provideimages for stereo analysis by system 100 for navigating vehicle 200,while image capture device 126 may provide images for monocular analysisby system 100 to provide redundancy and validation of informationobtained based on images captured from image capture device 122 and/orimage capture device 124. That is, image capture device 126 (and acorresponding processing device) may be considered to provide aredundant sub-system for providing a check on the analysis derived fromimage capture devices 122 and 124 (e.g., to provide an automaticemergency braking (AEB) system).

One of skill in the art will recognize that the above cameraconfigurations, camera placements, number of cameras, camera locations,etc., are examples only. These components and others described relativeto the overall system may be assembled and used in a variety ofdifferent configurations without departing from the scope of thedisclosed embodiments. Further details regarding usage of a multi-camerasystem to provide driver assist and/or autonomous vehicle functionalityfollow below.

FIG. 4 is an exemplary functional block diagram of memory 140 and/or150, which may be stored/programmed with instructions for performing oneor more operations consistent with the disclosed embodiments. Althoughthe following refers to memory 140, one of skill in the art willrecognize that instructions may be stored in memory 140 and/or 150.

As shown in FIG. 4, memory 140 may store a monocular image analysismodule 402, a stereo image analysis module 404, a velocity andacceleration module 406, and a navigational response module 408. Thedisclosed embodiments are not limited to any particular configuration ofmemory 140. Further, application processor 180 and/or image processor190 may execute the instructions stored in any of modules 402-408included in memory 140. One of skill in the art will understand thatreferences in the following discussions to processing unit 110 may referto application processor 180 and image processor 190 individually orcollectively. Accordingly, steps of any of the following processes maybe performed by one or more processing devices.

In one embodiment, monocular image analysis module 402 may storeinstructions (such as computer vision software) which, when executed byprocessing unit 110, performs monocular image analysis of a set ofimages acquired by one of image capture devices 122, 124, and 126. Insome embodiments, processing unit 110 may combine information from a setof images with additional sensory information (e.g., information fromradar) to perform the monocular image analysis. As described inconnection with FIGS. 5A-5D below, monocular image analysis module 402may include instructions for detecting a set of features within the setof images, such as lane markings, vehicles, pedestrians, road signs,highway exit ramps, traffic lights, hazardous objects, and any otherfeature associated with an environment of a vehicle. Based on theanalysis, system 100 (e.g., via processing unit 110) may cause one ormore navigational responses in vehicle 200, such as a turn, a laneshift, a change in acceleration, and the like, as discussed below inconnection with navigational response module 408.

In one embodiment, stereo image analysis module 404 may storeinstructions (such as computer vision software) which, when executed byprocessing unit 110, performs stereo image analysis of first and secondsets of images acquired by a combination of image capture devicesselected from any of image capture devices 122, 124, and 126. In someembodiments, processing unit 110 may combine information from the firstand second sets of images with additional sensory information (e.g.,information from radar) to perform the stereo image analysis. Forexample, stereo image analysis module 404 may include instructions forperforming stereo image analysis based on a first set of images acquiredby image capture device 124 and a second set of images acquired by imagecapture device 126. As described in connection with FIG. 6 below, stereoimage analysis module 404 may include instructions for detecting a setof features within the first and second sets of images, such as lanemarkings, vehicles, pedestrians, road signs, highway exit ramps, trafficlights, hazardous objects, and the like. Based on the analysis,processing unit 110 may cause one or more navigational responses invehicle 200, such as a turn, a lane shift, a change in acceleration, andthe like, as discussed below in connection with navigational responsemodule 408.

In one embodiment, velocity and acceleration module 406 may storesoftware configured to analyze data received from one or more computingand electromechanical devices in vehicle 200 that are configured tocause a change in velocity and/or acceleration of vehicle 200. Forexample, processing unit 110 may execute instructions associated withvelocity and acceleration module 406 to calculate a target speed forvehicle 200 based on data derived from execution of monocular imageanalysis module 402 and/or stereo image analysis module 404. Such datamay include, for example, a target position, velocity, and/oracceleration, the position and/or speed of vehicle 200 relative to anearby vehicle, pedestrian, or road object, position information forvehicle 200 relative to lane markings of the road, and the like. Inaddition, processing unit 110 may calculate a target speed for vehicle200 based on sensory input (e.g., information from radar) and input fromother systems of vehicle 200, such as throttling system 220, brakingsystem 230, and/or steering system 240 of vehicle 200. Based on thecalculated target speed, processing unit 110 may transmit electronicsignals to throttling system 220, braking system 230, and/or steeringsystem 240 of vehicle 200 to trigger a change in velocity and/oracceleration by, for example, physically depressing the brake or easingup off the accelerator of vehicle 200.

In one embodiment, navigational response module 408 may store softwareexecutable by processing unit 110 to determine a desired navigationalresponse based on data derived from execution of monocular imageanalysis module 402 and/or stereo image analysis module 404. Such datamay include position and speed information associated with nearbyvehicles, pedestrians, and road objects, target position information forvehicle 200, and the like. Additionally, in some embodiments, thenavigational response may be based (partially or fully) on map data, apredetermined position of vehicle 200, and/or a relative velocity or arelative acceleration between vehicle 200 and one or more objectsdetected from execution of monocular image analysis module 402 and/orstereo image analysis module 404. Navigational response module 408 mayalso determine a desired navigational response based on sensory input(e.g., information from radar) and inputs from other systems of vehicle200, such as throttling system 220, braking system 230, and steeringsystem 240 of vehicle 200. Based on the desired navigational response,processing unit 110 may transmit electronic signals to throttling system220, braking system 230, and steering system 240 of vehicle 200 totrigger a desired navigational response by, for example, turning thesteering wheel of vehicle 200 to achieve a rotation of a predeterminedangle. In some embodiments, processing unit 110 may use the output ofnavigational response module 408 (e.g., the desired navigationalresponse) as an input to execution of velocity and acceleration module406 for calculating a change in speed of vehicle 200.

FIG. 5A is a flowchart showing an exemplary process 500A for causing oneor more navigational responses based on monocular image analysis,consistent with disclosed embodiments. At step 510, processing unit 110may receive a plurality of images via data interface 128 betweenprocessing unit 110 and image acquisition unit 120. For instance, acamera included in image acquisition unit 120 (such as image capturedevice 122 having field of view 202) may capture a plurality of imagesof an area forward of vehicle 200 (or to the sides or rear of a vehicle,for example) and transmit them over a data connection (e.g., digital,wired, USB, wireless, Bluetooth, etc.) to processing unit 110.Processing unit 110 may execute monocular image analysis module 402 toanalyze the plurality of images at step 520, as described in furtherdetail in connection with FIGS. 5B-5D below. By performing the analysis,processing unit 110 may detect a set of features within the set ofimages, such as lane markings, vehicles, pedestrians, road signs,highway exit ramps, traffic lights, and the like.

Processing unit 110 may also execute monocular image analysis module 402to detect various road hazards at step 520, such as, for example, partsof a truck tire, fallen road signs, loose cargo, small animals, and thelike. Road hazards may vary in structure, shape, size, and color, whichmay make detection of such hazards more challenging. In someembodiments, processing unit 110 may execute monocular image analysismodule 402 to perform multi-frame analysis on the plurality of images todetect road hazards. For example, processing unit 110 may estimatecamera motion between consecutive image frames and calculate thedisparities in pixels between the frames to construct a 3D-map of theroad. Processing unit 110 may then use the 3D-map to detect the roadsurface, as well as hazards existing above the road surface.

At step 530, processing unit 110 may execute navigational responsemodule 408 to cause one or more navigational responses in vehicle 200based on the analysis performed at step 520 and the techniques asdescribed above in connection with FIG. 4. Navigational responses mayinclude, for example, a turn, a lane shift, a change in acceleration,and the like. In some embodiments, processing unit 110 may use dataderived from execution of velocity and acceleration module 406 to causethe one or more navigational responses. Additionally, multiplenavigational responses may occur simultaneously, in sequence, or anycombination thereof. For instance, processing unit 110 may cause vehicle200 to shift one lane over and then accelerate by, for example,sequentially transmitting control signals to steering system 240 andthrottling system 220 of vehicle 200. Alternatively, processing unit 110may cause vehicle 200 to brake while at the same time shifting lanes by,for example, simultaneously transmitting control signals to brakingsystem 230 and steering system 240 of vehicle 200.

FIG. 5B is a flowchart showing an exemplary process 500B for detectingone or more vehicles and/or pedestrians in a set of images, consistentwith disclosed embodiments. Processing unit 110 may execute monocularimage analysis module 402 to implement process 500B. At step 540,processing unit 110 may determine a set of candidate objectsrepresenting possible vehicles and/or pedestrians. For example,processing unit 110 may scan one or more images, compare the images toone or more predetermined patterns, and identify within each imagepossible locations that may contain objects of interest (e.g., vehicles,pedestrians, or portions thereof). The predetermined patterns may bedesigned in such a way to achieve a high rate of “false hits” and a lowrate of “misses.” For example, processing unit 110 may use a lowthreshold of similarity to predetermined patterns for identifyingcandidate objects as possible vehicles or pedestrians. Doing so mayallow processing unit 110 to reduce the probability of missing (e.g.,not identifying) a candidate object representing a vehicle orpedestrian.

At step 542, processing unit 110 may filter the set of candidate objectsto exclude certain candidates (e.g., irrelevant or less relevantobjects) based on classification criteria. Such criteria may be derivedfrom various properties associated with object types stored in adatabase (e.g., a database stored in memory 140). Properties may includeobject shape, dimensions, texture, position (e.g., relative to vehicle200), and the like. Thus, processing unit 110 may use one or more setsof criteria to reject false candidates from the set of candidateobjects.

At step 544, processing unit 110 may analyze multiple frames of imagesto determine whether objects in the set of candidate objects representvehicles and/or pedestrians. For example, processing unit 110 may tracka detected candidate object across consecutive frames and accumulateframe-by-frame data associated with the detected object (e.g., size,position relative to vehicle 200, etc.). Additionally, processing unit110 may estimate parameters for the detected object and compare theobject's frame-by-frame position data to a predicted position.

At step 546, processing unit 110 may construct a set of measurements forthe detected objects. Such measurements may include, for example,position, velocity, and acceleration values (relative to vehicle 200)associated with the detected objects. In some embodiments, processingunit 110 may construct the measurements based on estimation techniquesusing a series of time-based observations such as Kalman filters orlinear quadratic estimation (LQE), and/or based on available modelingdata for different object types (e.g., cars, trucks, pedestrians,bicycles, road signs, etc.). The Kalman filters may be based on ameasurement of an object's scale, where the scale measurement isproportional to a time to collision (e.g., the amount of time forvehicle 200 to reach the object). Thus, by performing steps 540-546,processing unit 110 may identify vehicles and pedestrians appearingwithin the set of captured images and derive information (e.g.,position, speed, size) associated with the vehicles and pedestrians.Based on the identification and the derived information, processing unit110 may cause one or more navigational responses in vehicle 200, asdescribed in connection with FIG. 5A, above.

At step 548, processing unit 110 may perform an optical flow analysis ofone or more images to reduce the probabilities of detecting a “falsehit” and missing a candidate object that represents a vehicle orpedestrian. The optical flow analysis may refer to, for example,analyzing motion patterns relative to vehicle 200 in the one or moreimages associated with other vehicles and pedestrians, and that aredistinct from road surface motion. Processing unit 110 may calculate themotion of candidate objects by observing the different positions of theobjects across multiple image frames, which are captured at differenttimes. Processing unit 110 may use the position and time values asinputs into mathematical models for calculating the motion of thecandidate objects. Thus, optical flow analysis may provide anothermethod of detecting vehicles and pedestrians that are nearby vehicle200. Processing unit 110 may perform optical flow analysis incombination with steps 540-546 to provide redundancy for detectingvehicles and pedestrians and increase the reliability of system 100.

FIG. 5C is a flowchart showing an exemplary process 500C for detectingroad marks and/or lane geometry information in a set of images,consistent with disclosed embodiments. Processing unit 110 may executemonocular image analysis module 402 to implement process 500C. At step550, processing unit 110 may detect a set of objects by scanning one ormore images. To detect segments of lane markings, lane geometryinformation, and other pertinent road marks, processing unit 110 mayfilter the set of objects to exclude those determined to be irrelevant(e.g., minor potholes, small rocks, etc.). At step 552, processing unit110 may group together the segments detected in step 550 belonging tothe same road mark or lane mark. Based on the grouping, processing unit110 may develop a model to represent the detected segments, such as amathematical model.

At step 554, processing unit 110 may construct a set of measurementsassociated with the detected segments. In some embodiments, processingunit 110 may create a projection of the detected segments from the imageplane onto the real-world plane. The projection may be characterizedusing a 3rd-degree polynomial having coefficients corresponding tophysical properties such as the position, slope, curvature, andcurvature derivative of the detected road. In generating the projection,processing unit 110 may take into account changes in the road surface,as well as pitch and roll rates associated with vehicle 200. Inaddition, processing unit 110 may model the road elevation by analyzingposition and motion cues present on the road surface. Further,processing unit 110 may estimate the pitch and roll rates associatedwith vehicle 200 by tracking a set of feature points in the one or moreimages.

At step 556, processing unit 110 may perform multi-frame analysis by,for example, tracking the detected segments across consecutive imageframes and accumulating frame-by-frame data associated with detectedsegments. As processing unit 110 performs multi-frame analysis, the setof measurements constructed at step 554 may become more reliable andassociated with an increasingly higher confidence level. Thus, byperforming steps 550-556, processing unit 110 may identify road marksappearing within the set of captured images and derive lane geometryinformation. Based on the identification and the derived information,processing unit 110 may cause one or more navigational responses invehicle 200, as described in connection with FIG. 5A, above.

At step 558, processing unit 110 may consider additional sources ofinformation to further develop a safety model for vehicle 200 in thecontext of its surroundings. Processing unit 110 may use the safetymodel to define a context in which system 100 may execute autonomouscontrol of vehicle 200 in a safe manner. To develop the safety model, insome embodiments, processing unit 110 may consider the position andmotion of other vehicles, the detected road edges and barriers, and/orgeneral road shape descriptions extracted from map data (such as datafrom map database 160). By considering additional sources ofinformation, processing unit 110 may provide redundancy for detectingroad marks and lane geometry and increase the reliability of system 100.

FIG. 5D is a flowchart showing an exemplary process 500D for detectingtraffic lights in a set of images, consistent with disclosedembodiments. Processing unit 110 may execute monocular image analysismodule 402 to implement process 500D. At step 560, processing unit 110may scan the set of images and identify objects appearing at locationsin the images likely to contain traffic lights. For example, processingunit 110 may filter the identified objects to construct a set ofcandidate objects, excluding those objects unlikely to correspond totraffic lights. The filtering may be done based on various propertiesassociated with traffic lights, such as shape, dimensions, texture,position (e.g., relative to vehicle 200), and the like. Such propertiesmay be based on multiple examples of traffic lights and traffic controlsignals and stored in a database. In some embodiments, processing unit110 may perform multi-frame analysis on the set of candidate objectsreflecting possible traffic lights. For example, processing unit 110 maytrack the candidate objects across consecutive image frames, estimatethe real-world position of the candidate objects, and filter out thoseobjects that are moving (which are unlikely to be traffic lights). Insome embodiments, processing unit 110 may perform color analysis on thecandidate objects and identify the relative position of the detectedcolors appearing inside possible traffic lights.

At step 562, processing unit 110 may analyze the geometry of a junction.The analysis may be based on any combination of: (i) the number of lanesdetected on either side of vehicle 200, (ii) markings (such as arrowmarks) detected on the road, and (iii) descriptions of the junctionextracted from map data (such as data from map database 160). Processingunit 110 may conduct the analysis using information derived fromexecution of monocular analysis module 402. In addition, Processing unit110 may determine a correspondence between the traffic lights detectedat step 560 and the lanes appearing near vehicle 200.

As vehicle 200 approaches the junction, at step 564, processing unit 110may update the confidence level associated with the analyzed junctiongeometry and the detected traffic lights. For instance, the number oftraffic lights estimated to appear at the junction as compared with thenumber actually appearing at the junction may impact the confidencelevel. Thus, based on the confidence level, processing unit 110 maydelegate control to the driver of vehicle 200 in order to improve safetyconditions. By performing steps 560-564, processing unit 110 mayidentify traffic lights appearing within the set of captured images andanalyze junction geometry information. Based on the identification andthe analysis, processing unit 110 may cause one or more navigationalresponses in vehicle 200, as described in connection with FIG. 5A,above.

FIG. 5E is a flowchart showing an exemplary process 500E for causing oneor more navigational responses in vehicle 200 based on a vehicle path,consistent with the disclosed embodiments. At step 570, processing unit110 may construct an initial vehicle path associated with vehicle 200.The vehicle path may be represented using a set of points expressed incoordinates (x, z), and the distance d_(i) between two points in the setof points may fall in the range of 1 to 5 meters. In one embodiment,processing unit 110 may construct the initial vehicle path using twopolynomials, such as left and right road polynomials. Processing unit110 may calculate the geometric midpoint between the two polynomials andoffset each point included in the resultant vehicle path by apredetermined offset (e.g., a smart lane offset), if any (an offset ofzero may correspond to travel in the middle of a lane). The offset maybe in a direction perpendicular to a segment between any two points inthe vehicle path. In another embodiment, processing unit 110 may use onepolynomial and an estimated lane width to offset each point of thevehicle path by half the estimated lane width plus a predeterminedoffset (e.g., a smart lane offset).

At step 572, processing unit 110 may update the vehicle path constructedat step 570. Processing unit 110 may reconstruct the vehicle pathconstructed at step 570 using a higher resolution, such that thedistance d_(k) between two points in the set of points representing thevehicle path is less than the distance d_(i) described above. Forexample, the distance d_(k) may fall in the range of 0.1 to 0.3 meters.Processing unit 110 may reconstruct the vehicle path using a parabolicspline algorithm, which may yield a cumulative distance vector Scorresponding to the total length of the vehicle path (i.e., based onthe set of points representing the vehicle path).

At step 574, processing unit 110 may determine a look-ahead point(expressed in coordinates as (x_(l), z_(l))) based on the updatedvehicle path constructed at step 572. Processing unit 110 may extractthe look-ahead point from the cumulative distance vector 5, and thelook-ahead point may be associated with a look-ahead distance andlook-ahead time. The look-ahead distance, which may have a lower boundranging from 10 to 20 meters, may be calculated as the product of thespeed of vehicle 200 and the look-ahead time. For example, as the speedof vehicle 200 decreases, the look-ahead distance may also decrease(e.g., until it reaches the lower bound). The look-ahead time, which mayrange from 0.5 to 1.5 seconds, may be inversely proportional to the gainof one or more control loops associated with causing a navigationalresponse in vehicle 200, such as the heading error tracking controlloop. For example, the gain of the heading error tracking control loopmay depend on the bandwidth of a yaw rate loop, a steering actuatorloop, car lateral dynamics, and the like. Thus, the higher the gain ofthe heading error tracking control loop, the lower the look-ahead time.

At step 576, processing unit 110 may determine a heading error and yawrate command based on the look-ahead point determined at step 574.Processing unit 110 may determine the heading error by calculating thearctangent of the look-ahead point, e.g., arctan (x_(l)/z_(l)).Processing unit 110 may determine the yaw rate command as the product ofthe heading error and a high-level control gain. The high-level controlgain may be equal to: (2/look-ahead time), if the look-ahead distance isnot at the lower bound. Otherwise, the high-level control gain may beequal to: (2*speed of vehicle 200/look-ahead distance).

FIG. 5F is a flowchart showing an exemplary process 500F for determiningwhether a leading vehicle is changing lanes, consistent with thedisclosed embodiments. At step 580, processing unit 110 may determinenavigation information associated with a leading vehicle (e.g., avehicle traveling ahead of vehicle 200). For example, processing unit110 may determine the position, velocity (e.g., direction and speed),and/or acceleration of the leading vehicle, using the techniquesdescribed in connection with FIGS. 5A and 5B, above. Processing unit 110may also determine one or more road polynomials, a look-ahead point(associated with vehicle 200), and/or a snail trail (e.g., a set ofpoints describing a path taken by the leading vehicle), using thetechniques described in connection with FIG. 5E, above.

At step 582, processing unit 110 may analyze the navigation informationdetermined at step 580. In one embodiment, processing unit 110 maycalculate the distance between a snail trail and a road polynomial(e.g., along the trail). If the variance of this distance along thetrail exceeds a predetermined threshold (for example, 0.1 to 0.2 meterson a straight road, 0.3 to 0.4 meters on a moderately curvy road, and0.5 to 0.6 meters on a road with sharp curves), processing unit 110 maydetermine that the leading vehicle is likely changing lanes. In the casewhere multiple vehicles are detected traveling ahead of vehicle 200,processing unit 110 may compare the snail trails associated with eachvehicle. Based on the comparison, processing unit 110 may determine thata vehicle whose snail trail does not match with the snail trails of theother vehicles is likely changing lanes. Processing unit 110 mayadditionally compare the curvature of the snail trail (associated withthe leading vehicle) with the expected curvature of the road segment inwhich the leading vehicle is traveling. The expected curvature may beextracted from map data (e.g., data from map database 160), from roadpolynomials, from other vehicles' snail trails, from prior knowledgeabout the road, and the like. If the difference in curvature of thesnail trail and the expected curvature of the road segment exceeds apredetermined threshold, processing unit 110 may determine that theleading vehicle is likely changing lanes.

In another embodiment, processing unit 110 may compare the leadingvehicle's instantaneous position with the look-ahead point (associatedwith vehicle 200) over a specific period of time (e.g., 0.5 to 1.5seconds). If the distance between the leading vehicle's instantaneousposition and the look-ahead point varies during the specific period oftime, and the cumulative sum of variation exceeds a predeterminedthreshold (for example, 0.3 to 0.4 meters on a straight road, 0.7 to 0.8meters on a moderately curvy road, and 1.3 to 1.7 meters on a road withsharp curves), processing unit 110 may determine that the leadingvehicle is likely changing lanes. In another embodiment, processing unit110 may analyze the geometry of the snail trail by comparing the lateraldistance traveled along the trail with the expected curvature of thesnail trail. The expected radius of curvature may be determinedaccording to the calculation: (δ_(z) ²+δ_(x) ²)/2/(δ_(x)), where δ_(x)represents the lateral distance traveled and δ_(z) represents thelongitudinal distance traveled. If the difference between the lateraldistance traveled and the expected curvature exceeds a predeterminedthreshold (e.g., 500 to 700 meters), processing unit 110 may determinethat the leading vehicle is likely changing lanes. In anotherembodiment, processing unit 110 may analyze the position of the leadingvehicle. If the position of the leading vehicle obscures a roadpolynomial (e.g., the leading vehicle is overlaid on top of the roadpolynomial), then processing unit 110 may determine that the leadingvehicle is likely changing lanes. In the case where the position of theleading vehicle is such that, another vehicle is detected ahead of theleading vehicle and the snail trails of the two vehicles are notparallel, processing unit 110 may determine that the (closer) leadingvehicle is likely changing lanes.

At step 584, processing unit 110 may determine whether or not leadingvehicle 200 is changing lanes based on the analysis performed at step582. For example, processing unit 110 may make the determination basedon a weighted average of the individual analyses performed at step 582.Under such a scheme, for example, a decision by processing unit 110 thatthe leading vehicle is likely changing lanes based on a particular typeof analysis may be assigned a value of “1” (and “0” to represent adetermination that the leading vehicle is not likely changing lanes).Different analyses performed at step 582 may be assigned differentweights, and the disclosed embodiments are not limited to any particularcombination of analyses and weights.

FIG. 6 is a flowchart showing an exemplary process 600 for causing oneor more navigational responses based on stereo image analysis,consistent with disclosed embodiments. At step 610, processing unit 110may receive a first and second plurality of images via data interface128. For example, cameras included in image acquisition unit 120 (suchas image capture devices 122 and 124 having fields of view 202 and 204)may capture a first and second plurality of images of an area forward ofvehicle 200 and transmit them over a digital connection (e.g., USB,wireless, Bluetooth, etc.) to processing unit 110. In some embodiments,processing unit 110 may receive the first and second plurality of imagesvia two or more data interfaces. The disclosed embodiments are notlimited to any particular data interface configurations or protocols.

At step 620, processing unit 110 may execute stereo image analysismodule 404 to perform stereo image analysis of the first and secondplurality of images to create a 3D map of the road in front of thevehicle and detect features within the images, such as lane markings,vehicles, pedestrians, road signs, highway exit ramps, traffic lights,road hazards, and the like. Stereo image analysis may be performed in amanner similar to the steps described in connection with FIGS. 5A-5D,above. For example, processing unit 110 may execute stereo imageanalysis module 404 to detect candidate objects (e.g., vehicles,pedestrians, road marks, traffic lights, road hazards, etc.) within thefirst and second plurality of images, filter out a subset of thecandidate objects based on various criteria, and perform multi-frameanalysis, construct measurements, and determine a confidence level forthe remaining candidate objects. In performing the steps above,processing unit 110 may consider information from both the first andsecond plurality of images, rather than information from one set ofimages alone. For example, processing unit 110 may analyze thedifferences in pixel-level data (or other data subsets from among thetwo streams of captured images) for a candidate object appearing in boththe first and second plurality of images. As another example, processingunit 110 may estimate a position and/or velocity of a candidate object(e.g., relative to vehicle 200) by observing that the object appears inone of the plurality of images but not the other or relative to otherdifferences that may exist relative to objects appearing if the twoimage streams. For example, position, velocity, and/or accelerationrelative to vehicle 200 may be determined based on trajectories,positions, movement characteristics, etc. of features associated with anobject appearing in one or both of the image streams.

At step 630, processing unit 110 may execute navigational responsemodule 408 to cause one or more navigational responses in vehicle 200based on the analysis performed at step 620 and the techniques asdescribed above in connection with FIG. 4. Navigational responses mayinclude, for example, a turn, a lane shift, a change in acceleration, achange in velocity, braking, and the like. In some embodiments,processing unit 110 may use data derived from execution of velocity andacceleration module 406 to cause the one or more navigational responses.Additionally, multiple navigational responses may occur simultaneously,in sequence, or any combination thereof.

FIG. 7 is a flowchart showing an exemplary process 700 for causing oneor more navigational responses based on an analysis of three sets ofimages, consistent with disclosed embodiments. At step 710, processingunit 110 may receive a first, second, and third plurality of images viadata interface 128. For instance, cameras included in image acquisitionunit 120 (such as image capture devices 122, 124, and 126 having fieldsof view 202, 204, and 206) may capture a first, second, and thirdplurality of images of an area forward and/or to the side of vehicle 200and transmit them over a digital connection (e.g., USB, wireless,Bluetooth, etc.) to processing unit 110. In some embodiments, processingunit 110 may receive the first, second, and third plurality of imagesvia three or more data interfaces. For example, each of image capturedevices 122, 124, 126 may have an associated data interface forcommunicating data to processing unit 110. The disclosed embodiments arenot limited to any particular data interface configurations orprotocols.

At step 720, processing unit 110 may analyze the first, second, andthird plurality of images to detect features within the images, such aslane markings, vehicles, pedestrians, road signs, highway exit ramps,traffic lights, road hazards, and the like. The analysis may beperformed in a manner similar to the steps described in connection withFIGS. 5A-5D and 6, above. For instance, processing unit 110 may performmonocular image analysis (e.g., via execution of monocular imageanalysis module 402 and based on the steps described in connection withFIGS. 5A-5D, above) on each of the first, second, and third plurality ofimages. Alternatively, processing unit 110 may perform stereo imageanalysis (e.g., via execution of stereo image analysis module 404 andbased on the steps described in connection with FIG. 6, above) on thefirst and second plurality of images, the second and third plurality ofimages, and/or the first and third plurality of images. The processedinformation corresponding to the analysis of the first, second, and/orthird plurality of images may be combined. In some embodiments,processing unit 110 may perform a combination of monocular and stereoimage analyses. For example, processing unit 110 may perform monocularimage analysis (e.g., via execution of monocular image analysis module402) on the first plurality of images and stereo image analysis (e.g.,via execution of stereo image analysis module 404) on the second andthird plurality of images. The configuration of image capture devices122, 124, and 126—including their respective locations and fields ofview 202, 204, and 206—may influence the types of analyses conducted onthe first, second, and third plurality of images. The disclosedembodiments are not limited to a particular configuration of imagecapture devices 122, 124, and 126, or the types of analyses conducted onthe first, second, and third plurality of images.

In some embodiments, processing unit 110 may perform testing on system100 based on the images acquired and analyzed at steps 710 and 720. Suchtesting may provide an indicator of the overall performance of system100 for certain configurations of image capture devices 122, 124, and126. For example, processing unit 110 may determine the proportion of“false hits” (e.g., cases where system 100 incorrectly determined thepresence of a vehicle or pedestrian) and “misses.”

At step 730, processing unit 110 may cause one or more navigationalresponses in vehicle 200 based on information derived from two of thefirst, second, and third plurality of images. Selection of two of thefirst, second, and third plurality of images may depend on variousfactors, such as, for example, the number, types, and sizes of objectsdetected in each of the plurality of images. Processing unit 110 mayalso make the selection based on image quality and resolution, theeffective field of view reflected in the images, the number of capturedframes, the extent to which one or more objects of interest actuallyappear in the frames (e.g., the percentage of frames in which an objectappears, the proportion of the object that appears in each such frame,etc.), and the like.

In some embodiments, processing unit 110 may select information derivedfrom two of the first, second, and third plurality of images bydetermining the extent to which information derived from one imagesource is consistent with information derived from other image sources.For example, processing unit 110 may combine the processed informationderived from each of image capture devices 122, 124, and 126 (whether bymonocular analysis, stereo analysis, or any combination of the two) anddetermine visual indicators (e.g., lane markings, a detected vehicle andits location and/or path, a detected traffic light, etc.) that areconsistent across the images captured from each of image capture devices122, 124, and 126. Processing unit 110 may also exclude information thatis inconsistent across the captured images (e.g., a vehicle changinglanes, a lane model indicating a vehicle that is too close to vehicle200, etc.). Thus, processing unit 110 may select information derivedfrom two of the first, second, and third plurality of images based onthe determinations of consistent and inconsistent information.

Navigational responses may include, for example, a turn, a lane shift, achange in acceleration, and the like. Processing unit 110 may cause theone or more navigational responses based on the analysis performed atstep 720 and the techniques as described above in connection with FIG.4. Processing unit 110 may also use data derived from execution ofvelocity and acceleration module 406 to cause the one or morenavigational responses. In some embodiments, processing unit 110 maycause the one or more navigational responses based on a relativeposition, relative velocity, and/or relative acceleration betweenvehicle 200 and an object detected within any of the first, second, andthird plurality of images. Multiple navigational responses may occursimultaneously, in sequence, or any combination thereof.

Lane End Recognition System

System 100 may recognize when a driving lane will end and, when an endis detected, make a responsive navigational change in the course ofvehicle 200, such as a lane change. A “lane” may refer to a designatedor intended travel path of a vehicle and may have marked (e.g., lines ona road) or unmarked boundaries (e.g., an edge of a road, a road barrier,guard rail, parked vehicles, etc.), or constraints. Rather than relyingon map data to make a lane end determination, the system may operatebased solely on visual information. In some cases, however, the systemmay rely upon a combination of map data and visual information relativeto a lane of travel. Further, while the system may recognize an end of alane by direct observation of the lane ending, the system may recognizean approaching lane end sooner by, for example, observing andrecognizing road sign alerts that provide information regarding a laneending.

FIG. 8 illustrates a vehicle 200 traveling on a roadway 800 in which thedisclosed systems and methods for recognizing the end of a travel laneand operating vehicle 200 within the lanes may be used. While vehicle200 is depicted as being equipped with image capture devices 122 and124, more or fewer cameras may be employed on any particular vehicle200. As shown, roadway 800 may be subdivided into lanes, such as lanes810 and 820. Lanes 810 and 820 are shown as examples; a given roadway800 may have additional lanes based on the size and nature of theroadway, for example, an interstate highway. In the example of FIG. 8,vehicle 200 is traveling in lane 810, and lane 810 can be seen to beending shortly. In response, system 100 may, as discussed in detailbelow, cause vehicle 200 to shift to lane 820.

Processing unit 110 may be configured to determine one or more laneconstraints associated with each of lanes 810 and 820 based on aplurality of images acquired by image capture device 122-126 thatprocessing unit 110 may receive via data interface 128. According tosome embodiments, the lane constraints may be identified by visible laneboundaries, such as dashed or solid lines marked on a road surface.Additionally or alternatively, the lane constraints may include an edgeof a road surface or a barrier. Additionally or alternatively, the laneconstraints may include markers (e.g., Botts' dots). According to someembodiments, processing unit 110 (via lane constraint module 910 and/orlane positioning module 920, described in detail below) may determineconstraints associated with lanes 810/820 by identifying a midpoint of aroad surface width, such as the entirety of roadway 800 or a midpoint ofone of lanes 810/820. Processing unit 110 may identify lane constraintsin alternative manners, such as by estimation or extrapolation based onknown roadway parameters when, for example, lines designating road lanessuch as lanes 810/820 are not painted or otherwise labeled.

Detection of the constraints and pathways of roadway 800 and constituentlanes 810/820 may include processing unit 110 determining their 3Dmodels via a camera coordinate system. For example, the 3D models oflanes 810/820 may be described by a third-degree polynomial. In additionto 3D modeling of travel lanes, processing unit 110 may performmulti-frame estimation of host motion parameters, such as the speed, yawand pitch rates, and acceleration of vehicle 200. Processing unit 110may receive information pertaining to these parameters from, forexample, speed sensors, multi-axis accelerometers, etc., included invehicle 200. Optionally, processing unit 110 may detect static andmoving vehicles and their position, heading, speed, and acceleration,all relative to vehicle 200, which will be described below inassociation with FIG. 10B. Processing unit 110 may further determine aroad elevation model to transform all of the information acquired fromthe plurality of images into 3D space.

Generally, as a default condition, vehicle 200 may travel relativelycentered within lanes 810/820, so that distances x₁ and x₂ from vehicle200 to the lane constraints associated with lanes 810/820 depicted inFIG. 8 are equal or close to equal. However, in some circumstances,environmental factors may make this undesirable or unfeasible, such aswhen objects or structures are present on one side of the road. Thus,there may be circumstances in which it may be desirable or advantageousfor vehicle 200 to be closer to one lane constraint or the other (i.e.x₁>x₂ or vice versa), particularly if the lane is determined to beending, as will be discussed below in association with FIGS. 10-11.

FIG. 9 is an exemplary block diagram of memory 140 and/or 150, which maystore instructions for performing one or more operations consistent withdisclosed embodiments. As illustrated in FIG. 9, memory 140 may storeone or more modules for performing the lane end recognition andresponses described herein. For example, memory 140 may store a laneconstraint module 910, a lane positioning module 920, and an actionmodule 930. Application processor 180 and/or image processor 190 mayexecute the instructions stored in any of modules 910-930 included inmemory 140. One of skill in the art will understand that references inthe following discussions to processing unit 110 may refer toapplication processor 180 and image processor 190 individually orcollectively. Accordingly, steps of any of the following processes maybe performed by one or more processing devices.

Lane constraint module 910 may store instructions which, when executedby processing unit 110, may detect and define boundaries associated withlanes such as lanes 810 and 820. In some embodiments the detectedboundaries may be artificial, for example, painted lines or reflectors.In other embodiments, the detected boundaries may simply include thenatural curvature of the road or a termination of a lane or roadsurface. Lane offset module 910 may process the plurality of imagesreceived from at least one image capture device 122-124 to detect laneconstraints. As discussed above, this may include identifying paintedlane lines and/or measuring a midpoint of a road surface. Additionally,lane constraint module 910 may detect the course and/or constraint oftravel lanes such as lanes 810 and 820 by extracting information fromone or more road signs within the field of view of image capturedevice(s) 122-124. Examples of these road signs may include, but are notlimited to, signs containing textual messages such as “Lane Ends—MergeLeft” or signs containing visual messages, such as a visualrepresentation of a lane ending.

Lane positioning module 920 may store instructions which, when executedby processing unit 110, may assist system 100 in determining a positionof vehicle 200. In some embodiments, determining the position of vehicle200 may include determining, based on at least one indicator of vehicleposition, a distance from the vehicle to one or more lane constraintsassociated with the current lane in which the vehicle is traveling. Suchdeterminations of the position of vehicle 200 may be based at least inpart on an analysis of image data captured by one or more of imagecapture devices 122-126. In some embodiments, the image data may besupplemented by or substituted with map data (e.g., GPS information)and/or data from other sensors (e.g., laser range finders, etc.). Inthese embodiments, the distance may be measured relative to laneconstraints identified by lane constraint module 910. For example, thedistance to the one or more lane constraints may specify the distance toan end of a lane or an area in which a lane has partially ended (e.g.,for a lane that includes a merging zone that diminishes over distanceuntil the lane has ended). Lane positioning module 920 may process theplurality of images to detect features of the roadway on which vehicle200 is traveling, such as roadway 800, in order to determine theposition of vehicle 200. Additionally or alternatively, lane offsetmodule 920 may receive information from other modules (includingposition sensor 130) or other systems indicative of the presence ofother features in the image data, such as additional vehicles, curvatureof the road, etc. Lane positioning module 920 may periodically updatethe positional status of the vehicle using coordinate systems or othernotations of position (e.g., map data from a mapping application, GPSinformation, etc.). The status may be stored within the system, such aswithin memory 140/150. Lane positioning module 920 may update thepositional status while vehicle 200 is moving at set timepoints, such aseach minute, after a predetermined number of minutes, each hour, etc.Alternatively, lane positioning module 920 may update the positionalstatus when certain vehicle operations are detected, such as changinglanes, accelerating, decelerating, or other steering operations.

Action module 930 may store instructions which, when executed byprocessing unit 110, may assist system 100 in taking one or more actionsrelative to the operation of vehicle 200 based on information receivedfrom one or more sources, such as position sensor 130, lane constraintmodule 910, or lane positioning module 920. In some embodiments, actionmodule 930 may receive information indicating that the current travellane of vehicle 200 is ending. Action module 930 may then perform one ormore actions affecting the operational status of vehicle 200, such ascausing system 100 to provide control signals to one or more ofthrottling system 220, braking system 230, and steering system 240 tonavigate vehicle 200 (e.g., by causing an acceleration, a turn, a laneshift, etc.). For example, in some embodiments, action module 930 maysend an instruction to steering system 240, and steering system 240 mayexecute the instruction to steer the vehicle into a new lane of travel.Additionally or alternatively, action module 300 may send instructionsto other systems associated with vehicle 200, such as braking system230, turn signals, throttling system 220, etc. In some embodiments,action module 930 may instead provide a human operator of the vehiclewith audio, visual, haptic, or tactile feedback representative of theinformation gathered from the relevant systems and/or sensors. The humanoperator may then act on this feedback to steer the vehicle to adifferent lane.

FIG. 10A provides an annotated view of the situation depicted in FIG. 8.Vehicle 200 is once again traveling in lane 810 of roadway 800, which isabout to end. Vehicle 200 is again equipped with image capture devices122 and 124; more or fewer devices may be associated with any particularvehicle 200.

To the right side of roadway 800 are road signs 1002 and 1004. Thesesigns are exemplary, and not intended to be limiting. Road sign 1002presents a visual depiction of a lane ending. This sign is familiar tomany drivers, indicating a need to merge. Sign 1004, typically placedcloser to the actual end of the lane, informs drivers that the “LANEENDS—MERGE LEFT.” Consistent with disclosed embodiments, system 100 maybe configured to extract information from road signs such as signs 1002and 1004 for purposes of determining if and when the current travel lanethat a vehicle 200 is traveling in is ending. For visual signs such assign 1002, image capture devices 122-126 may capture images of theentire face of the sign. Image processor 190 may be configured torecognize the image pattern of sign 1002, and may transmit informationto other modules or units within system 100 indicating that the signinstructs that the lane is ending. In some embodiments, constituentmodules or subsystems of system 100, such as memory 140/150, may beconfigured to contain lists or databases of typical road signs andpatterns for purposes of rapid, real-time detection and informationextraction. Sign 1004 contains only text, and thus its warning may beread by system 100 through optical character recognition (OCR)capability, which may be configured within processing unit 110, memory140/150, image processor 190, or any other constituent subsystem ormodule of system 100.

Extraction of information from road signs is one of several actions thatsystem 100 may be configured to execute for purposes of lanerecognition. In addition, system 100, via lane constraint module 910 andlane positioning module 920 within memory 140/150, may be configured toperform visual determinations of roadway 800 and make accompanyingdistance measurements based on at least one indicator of position of thevehicle. In the example of FIG. 10A, various potential measurements thatmay be used by system 100 are shown. System 100 may obtain themeasurements by, for example, analyzing one or more images acquired byone or more of image capture devices 122-126. Measurement 1006 mayrepresent the total width w of lane 810. Distance w may also be thoughtof as the sum of distances x₁ and x₂ from FIG. 8 plus the width ofvehicle 200. Measurement 1008 may represent the radius of curvature r ofroadway 800 as lane 810 comes to an end. A shorter radius r may indicatea more urgent need for system 100 to steer vehicle 200 into lane 820.Measurement 1010 may represent the distance d₁ from vehicle 200 to aconstraint associated with lane 810. This constraint may indicate thatlane 810 is ending. As discussed above, the constraint may be a paintedline, or may be the natural edge of roadway 800. Distance d₁ may also bemeasured to any portion in three-dimensional space of a barrier, such asa guardrail, wall, barrel, cone, or any other such object. Distance d₁and other such distances, such as x₁, x₂, and w, may be measured fromany portion of the interior or exterior vehicle 200, including but notlimited to the front of vehicle 200, a portion of vehicle 200 such as aheadlight or front license plate, a position as-installed of imagecapture devices 122 or 124, a determined centroid of vehicle 200, therear of vehicle 200, one or more windshields or mirrors associated withvehicle 200, wheels of vehicle 200, right or left sides or windows ofvehicle 200, a point associated with the roof of vehicle 200, or a pointassociated with the chassis of vehicle 200. As will be discussed belowin association with process 1100 of FIG. 11, system 100 may use one orboth of the extracted road sign information or the distances 1006-1010to determine if the current travel lane of vehicle 200 is ending.

FIG. 10B illustrates an additional visual determination and/or set ofdistance measurements that system 100 may use to determine if a currenttravel lane is ending on a roadway. In the example of FIG. 10B, thereare two vehicles 200 a and 200 b driving in lane 810 of roadway 800,which is ending. In some embodiments, vehicle 200 a may be equipped witha system 100 and may measure a distance d₂ to the back side of vehicle200 b. Distance d₂ may be measured to or from any of the portions ofvehicles 200 a and 200 b discussed previously. Alternatively, vehicle200 b may be equipped with the system 100, and may measure distance d₂backwards to vehicle 200 a. Both vehicles are depicted as being equippedwith systems 100 for purposes of FIG. 10B's exemplary illustration. Insome embodiments, the system 100 (on whichever of the two vehiclesillustrated in FIG. 10B) may be configured to make a visualdetermination of whether the current travel lane is ending based ondistance d₂. In these embodiments, system 100 may be configured torecognize that the lane is ending based on an abrupt shortening ofdistance d₂. Alternatively, system 100 may detect that distance d₂ hasacquired a lateral element instead of being a straight line extendingnormal to a front plane associated with vehicle 200 a. When configuredin this manner, measurements such as c₁-c₄ may be measured and used forpositional status determination by lane positioning module 920.Distances c₁-c₄ may represent the distance from a side of vehicle 200 aor 200 b to the edge of lane 810, be it lane constraint 1014 or thedashed center line in the middle of roadway 800 dividing lanes 810/820.In other embodiments, a distance may be measured to lane constraint 1012on the far side of lane 820 (not shown).

As in FIG. 8 and in FIG. 10A, lane 810 in FIG. 10B is ending, and bothvehicles 200 a and 200 b will need to merge into lane 820. FIG. 10Billustrates additional distances that system 100 may be configured tocalculate, extract, and/or analyze in a live driving situation. Inaddition to distances c₁-c₄ described above, system 100 may be furtherconfigured to calculate distances w₁ and w₂ and midpoint in of lane 810relative to one or more lane constraints associated with that lane. Whensummed together, distances w₁ and w₂ equal measurement 1006 w as shownin FIG. 10A. System 100, via lane positioning module 920, mayperiodically measure distances w₁ and w₂ and midpoint m of lane 810 overset timepoints. Should distances w₁ and w₂ shrink, and/or if midpoint mshifts to the right or left, system 100 may determine that lane 810 isending and that a merge to the other lane is needed.

FIG. 11 illustrates a process 1100 for detecting a lane ending for avehicle, consistent with disclosed embodiments. Steps of process 1100may be performed by one or more of processing unit 110, imageacquisition unit 120, position sensor 130, lane constraint module 910,lane positioning module 920, or action module 930.

At step 1110, process 1100 may include acquiring, using at least oneimage capture device 122, 124, and/or 126, at least one image of an areaforward of vehicle 200. The area may further include at least one roadsign providing information indicative of the lane ending, such as roadsigns 1002 and/or 1004. Processing unit 110 may receive the plurality ofimages from the image capture device(s) through data interface 128. Theat least one image may then be processed in real time by image processor190 of processing unit 110.

At step 1120, process 1100 may extract lane ending information from arepresentation of the at least one road sign included in the at leastone image. The extraction may be carried out by one or more of imageprocessor 190, lane constraint module 910, or lane positioning module920. A variety of lane ending information may be extracted from theimage data. For example, on roadways such as roadway 800 comprising aplurality of travel lanes (such as lanes 810 and 820), the extractedlane ending information may include an identification of which of thelanes from the plurality of lanes is ending. In other embodiments, theextracted lane ending information may include a distance to the laneending. Image processor 190, lane constraint module 910, and/or lanepositioning module 920 may identify one or more objects or laneconstraints to help determine the distance until the lane ends. Takingthe example of FIG. 10A, one or more images received from data interface128 may include road signs 1002 and 1004, and may factor in roadcurvature 1008 to determine the distance 1010 remaining in the lane. Insome embodiments, a value of curvature 1008 may itself independentlyconstitute the extracted lane ending information. As part of processingthe extracted information, system 100 may determine the distance fromvehicle 200 to one or more of the identified objects (e.g., laneconstraints) using position sensor 130. In some embodiments, system 100may determine the distance from vehicle 200 to one or more of theidentified objects (e.g., lane constraints) by analyzing one or moreimages acquired by one or more of image capture devices 122-126. In yetother embodiments, system 100 may determine the distance from vehicle200 to one or more of the identified objects (e.g., lane constraints) byusing a combination of data provided by position sensor 130 and dataobtained through an analysis of one or more images acquired by one ormore of image capture devices 122-126. At step 1130, process 1100 maydetermine a position of the vehicle, such as vehicle 200, based on atleast one indicator of position of the vehicle. The at least oneindicator of position of the vehicle may include information obtained bya direct measuring tool, physical device, or module, such as positionsensor 130. Alternatively or additionally, the at least one indicator ofposition of the vehicle may include information obtained by an indirectmeasurement. For example, in some embodiments the at least one indicatorof position may include a visual determination of which lane (such aslanes 810 and 820) among a plurality of lanes on a roadway 800 thatvehicle 200 may be traveling. In other embodiments, the at least oneindicator of position may include one or more visual determinations. Thevisual determinations may be directly determined by softwareinstructions executed by image acquisition unit 120 in the manner of a“range finder,” or may be determined from analysis of image datareceived from data interface 128 by image processor 190 and acquired byone or more of image capture devices 122-126. In some embodiments, theat least one indicator of position may include a visual determination ofa distance to the lane ending. This determination may, for example, bemade by lane constraint module 910 and lane positioning module 920 basedon a determined distance from vehicle 200 to one or more laneconstraints of roadway 800 associated with one or more lanes. In stillother embodiments, the at least one indicator of position may include avisual determination of a distance to a second vehicle. The secondvehicle may be traveling in front of vehicle 200. This is the scenariodescribed above in association with FIG. 10B. Once determined, thepositional information may be stored within system 100, such as inmemory 140/150, and may be periodically updated by lane positioningmodule 920 as discussed above in association with FIG. 9.

At step 1140, process 1100 may determine (based on the extracted laneinformation from step 1120 and the determined distance from the vehicleto the one or more roadway 800 lane constraints from step 1130), whetherthe current lane in which the vehicle is traveling is ending. Asdescribed generally above, system 100 may determine the ongoing statusof a travel lane, such as lanes 810 and 820, within a roadway 800. Forexample, system 100 may extract information from road signs ahead of thevehicle, such as road signs 1002 and 1004 indicating that the lane inwhich vehicle 200 is traveling in may be ending shortly. Additionally,measurements of distance may be made, using various elements of system100. For example, position sensor 130 and/or lane positioning module 920may send information to processing unit 110 regarding the relativeposition of vehicle 200 within lane 810/820, including how far vehicle200 is from the end of the lane. The end of the lane may be based on ameasured distance or a visual determination (e.g., by analyzing one ormore image images acquired by one or more of image capture devices122-126).

If processing unit 110 determines from the acquired information that thecurrent travel lane is not ending (Step 1140: NO), then process 1100 mayrevert back to the beginning, and system 100 may continue to monitor thestatus of vehicle 200 with respect to its current lane of travel. Thiscondition may result, for example, if a road sign indicates that a laneis ending due to road construction, but due to an overriding exception,however, such as weather, the day of the week, or other factors, noworkers or obstacles are actually blocking the lane. System 100 may alsodetermine, for example, via lane constraint module 910 that a perceivedshift in one or more lane constraints may be due to a new or temporaryroad pattern, or may comprise a natural curvature in the road. Ofcourse, the driver (if any) of vehicle 200 is still capable of changinglanes manually.

Alternatively, if system 100 determines that the current travel lane isending (Step 1140: YES), then process 1100 may proceed to step 1150. Atstep 1150, process 1100 may cause vehicle 200 to change lanes to anotherlane of roadway 800 that is not ending. Via action module 930,processing unit 110 may send an instruction to one or more systemsassociated with vehicle 200. For example, an instruction may be sent tosteering system 240, instructing vehicle 200 to merge left or right intothe next lane. Additional instructions may be sent; for example, imageprocessor 190 may determine from real-time image data that trafficprevents an immediate merge into the next lane. In these embodiments, aninstruction may first be sent to braking system 230 and/or the turnsignal control system, and vehicle 200 may decelerate or be brought to astop until the desired new lane is safe to merge into. At that time, theinstruction to steering system 220 may be made and the vehicle may turn.In some alternative embodiments, where vehicle 200 is partially or fullyunder the control of a human operator, step 1150 may instead comprisegenerating feedback to the operator informing that the travel lane isending and that a lane change is required. In some embodiments, anautomated system such as system 100 and its action module 930 may takeover operation of the car and cause the vehicle to change lanesautomatically as described above if the human operator does not respondto the feedback within a certain time period. The time period may bepredetermined, or may be calculated based on visual determinations ofthe distance to the end of the lane such as those described above inassociation with step 1130.

Detection of Low-Height Objects in a Roadway

System 100 may detect when an object is situated within a current laneof travel and, if system 100 determines that the object exceeds athreshold height, the system may steer an associated vehicle around orover the object. A “lane” may refer to a designated or intended travelpath of a vehicle and may have marked (e.g., lines on a road) orunmarked boundaries (e.g., an edge of a road, a road barrier, guardrail, parked vehicles, etc.), or constraints. System 100 may operate tomake these detections and determinations based solely on visualinformation acquired by one or more image acquisition units. In someembodiments, system 100 may be configured to detect objects in a roadwaythat project above the roadway surface by as little as 2 cm.

FIG. 12 illustrates a vehicle 200 traveling on a roadway 800 in whichthe disclosed systems and methods for detecting an object in the roadwaymay be used. Vehicle 200 is depicted as being equipped with imagecapture devices 122 and 124; more or fewer image capture devices(including cameras, for example) may be employed on any particularvehicle 200. As shown, roadway 800 may be subdivided into lanes, such aslanes 810 and 820. Lanes 810 and 820 are shown as examples; a givenroadway 800 may have additional lanes based on the size and nature ofthe roadway, for example, an interstate highway. In the example of FIG.12, vehicle 200 is traveling in lane 810, and two objects 1202 and 1204are located in lane 810. System 100 may, as discussed in detail below,cause vehicle 200 to steer around or over the object(s).

Processing unit 110 may be configured to determine one or more laneconstraints associated with each of lanes 810 and 820 based on aplurality of images acquired by image capture device 122-126 thatprocessing unit 110 may receive via data interface 128. According tosome embodiments, the lane constraints may be identified by visible laneboundaries, such as dashed or solid lines marked on a road surface.Additionally or alternatively, the lane constraints may include an edgeof a road surface or a barrier. Additionally or alternatively, the laneconstraints may include markers (e.g., Botts' dots). According to someembodiments, processing unit 110 (via lane positioning module 1320,described in detail below) may determine constraints associated withlanes 810/820 by identifying a midpoint of a road surface width, such asthe entirety of roadway 800 or a midpoint of one of lanes 810/820.Processing unit 110 may identify lane constraints in alternativemanners, such as by estimation or extrapolation based on known roadwayparameters when, for example, lines designating road lanes such as lanes810/820 are not painted or otherwise labeled.

Detection of the constraints and pathways of roadway 800 and constituentlanes 810/820 may include processing unit 110 determining their 3Dmodels via a camera coordinate system. For example, the 3D models oflanes 810/820 may be described by a third-degree polynomial. In additionto 3D modeling of travel lanes, processing unit 110 may performmulti-frame estimation of host motion parameters, such as the speed, yawand pitch rates, and/or acceleration of vehicle 200. Processing unit 110may detect static and moving objects and their position, all relative tovehicle 200, which will be described below in association with FIGS.14A, 14B, and 15. For example, processing unit 110 may use image data,GPS data, and/or data from sensors (e.g. position sensor 130, speedsensors, etc.) to determine the position of vehicle 200. Processing unit110 may further determine a road elevation model to transform a portionor all of the information acquired from the plurality of images into 3Dspace.

In the example of FIG. 12, several measurements are represented thatsystem 100 may take into account when determining potential actions totake using action module 1330, which will be described in detail below.Objects 1202 and 1204 are represented in the illustration of FIG. 12 asa cardboard box and a ladder, respectively. These are common items thatmay be found as obstacles in a typical roadway, for example, afterfalling from an unseen second vehicle. These objects are meant asexamples only. System 100 may be configured to detect the presence ofany object that may be found in a roadway, including but not limited tobranches, tires or portions thereof, scraps of metal or wood, speedbumps within a travel lane, a human or animal, a pipe, a piece ofanother vehicle, etc.

As part of a driver-assist object detection system consistent withdisclosed embodiments, system 100 may determine, for example, distancesI₁ and I₂, which may represent the distance of a surface of vehicle 200to various features of roadway 800 and/or lanes 810/820, such as laneconstraints, road signs, or any other such feature. Further, system 100may determine distances d₃ and d₄, which are distances measured from asurface of vehicle 200 to each of objects 1202 and 1204, respectively.Distances such as I₁, I₂, d₃, and d₄, may be measured from any portionof the interior or exterior vehicle 200, including but not limited tothe front of vehicle 200, a portion of vehicle 200 such as a headlightor front license plate, a position as-installed of image capture devices122 or 124, a determined centroid of vehicle 200, the rear of vehicle200, one or more windshields or mirrors associated with vehicle 200,wheels of vehicle 200, right or left sides or windows of vehicle 200, apoint associated with the roof of vehicle 200, or a point associatedwith the chassis of vehicle 200. Further detail on how these exemplarydistance measurements may be used to determine potential actions takenby system 100 will now be discussed in detail.

FIG. 13 is an exemplary block diagram of memory 140 and/or 150, whichmay store instructions for performing one or more operations consistentwith disclosed embodiments. As illustrated in FIG. 13, memory 140 maystore one or more modules for performing the object detection andresponses described herein. For example, memory 140 may store an objectdetection module 1310, a lane positioning module 1320, and an actionmodule 1330. Application processor 180 and/or image processor 190 mayexecute the instructions stored in any of modules 1310-1330 included inmemory 140. One of skill in the art will understand that references inthe following discussions to processing unit 110 may refer toapplication processor 180 and image processor 190 individually orcollectively. Accordingly, steps of any of the following processes maybe performed by one or more processing devices.

Object detection module 1310 may store instructions which, when executedby processing unit 110, may detect one or more objects found within aroadway 800, such as within lanes 810 and 820. As will be discussedbelow in association with FIGS. 14A, 14B, and 15, object detectionmodule 1310 may identify one or more objects within image data capturedby one or more of image capture devices 122-126 and determine that thoseobjects are resting on the roadway surface. Object detection module 1310may further determine a reference plane corresponding to the roadsurface, and using the reference plane, may determine dimensions andattributes of the identified object(s), such as their height, length,and width. The object detection process may further include identifyingpainted lane lines and/or measuring a midpoint of a road surface such asroadway 800, for purposes of determining the reference plane and fordetermining the location of the objects within the roadway/lanes. Insome embodiments, the field of view of the image capture device(s) usedto generate the image data may range from about 35 to about 55 degrees.

Lane positioning module 1320 may store instructions which, when executedby processing unit 110, may assist system 100 in determining a positionof vehicle 200. In some embodiments, determining the position of vehicle200 may include determining, based on at least one indicator of vehicleposition, a distance from the vehicle to one or more lane constraintsassociated with the current lane in which the vehicle is traveling. Inthese embodiments, the distance may also be measured relative toidentified objects detected by object detection module 1310. Consistentwith disclosed embodiments, the distance(s) measured to the one or morelane constraints may be used to determine if a change in a directionalcourse of vehicle 200 is required or recommended. Lane positioningmodule 1320 may process the plurality of images to detect features ofthe roadway on which vehicle 200 is traveling, such as roadway 800, inorder to determine the position of vehicle 200. Additionally oralternatively, lane positioning module 1320 may receive information fromother modules (including position sensor 130) or other systemsindicative of the presence of other features in the image data, such asadditional vehicles, curvature of the road, etc.

Action module 1330 may store instructions which, when executed byprocessing unit 110, may assist system 100 in taking one or more actionsrelative to the operation of vehicle 200 based on information receivedfrom one or more sources, such as position sensor 130, object detectionmodule 1310, or lane positioning module 1320. In some embodiments,action module 1330 may receive information (from, e.g., object detectionmodule 1310) indicating that one or more objects have been detected inroadway 800. The information may further include one or more dimensionsor attributes of the detected object, such as its height relative to theroadway surface. Action module 1330 may then perform one or more actionsaffecting the operational status of vehicle 200, such as causing system100 to provide control signals to one or more of throttling system 220,braking system 230, and steering system 240 to navigate vehicle 200(e.g., by causing an acceleration, a turn, a swerve, a lane shift,etc.). For example, in some embodiments, action module 1330 may send aninstruction to steering system 240, and steering system 240 may executethe instruction to steer the vehicle into a new lane of travel. In otherembodiments, action module 1330 may determine that a safer and/or bettercourse of action is to simply steer vehicle 200 such that it passes overtop of the object, assuming the height of the object is less than theroad clearance height of vehicle 200. Additionally or alternatively,action module 1330 may send instructions to other systems associatedwith vehicle 200, such as braking system 230, turn signals, throttlingsystem 220, etc. In some embodiments, action module 1330 may insteadprovide a human operator of the vehicle with audio, visual, or tactilefeedback representative of the information gathered from the relevantsystems and/or sensors. The human operator may then act on this feedbackto steer the vehicle to a different lane.

FIG. 14A illustrates examples of two images among the plurality ofcaptured images that may be acquired by system 100 via image capturedevices 122-126. In the example of FIG. 14A, image 1402 (top) may be thefirst image acquired, and image 1404 (bottom) may be a second imagetaken at a later time, such as a few seconds later. As can be seen inimages 1402 and 1404, a vehicle equipped with a system 100, such asvehicle 200 above, is driving on a roadway, such as roadway 800. Theroadway is aligned straight ahead of the vehicle, with the view of theroad ending at horizon 1406. Based on the view illustrated in image1402, the roadway may be cresting a hill. Any roadway configuration oralignment may be capable of being analyzed by system 100. As can be seenin both images 1402 and 1404, object 1202, representing a cardboard boxof initially unknown dimensions, is lying in the middle of the righttravel lane of the illustrated roadway.

In some embodiments, system 100 may be configured to analyze capturedimages, such as images 1402 and 1404, to determine a reference planecorresponding to a road plane, such as the roadway surface. This processwill be described in more detail below in association with FIG. 15 andprocess 1500. Briefly, system 100 (via processing unit 110 and imageprocessor 190) may determine the reference plane corresponding to theroadway surface as an average surface level associated with the roadsurface. This may account for variations in the roadway surface due tothe composition of the roadway (i.e., asphalt, concrete, or even gravel)or changes in the pitch and/or curvature of the roadway. Multiple imagesfrom among the plurality of images may be used to determine thereference plane. In the illustration of FIG. 14A, the determinedreference plane is represented by reference plane 1408.

System 100, via processing unit 110 and object detection module 1310,may locate one or more objects lying in the roadway, such as object1202. The detection may occur, for example, when the object appears inthe same location in multiple images among the captured plurality ofimages. Objects, such as object 1202, may be moving or stationary.Object 1202 is stationary in the roadway in the illustration of FIG.14A. The location(s) of the object(s) may be determined by objectdetection module 1310 by various methods. In some embodiments, thelocation of the target object(s) may be determined by GPS and/or othercoordinate-based systems. Further, object detection module 1310 may makea visual determination of the location of object 1202. In someembodiments, one or more of object detection module 1310, positionsensor 130, and image processor 190 may further determine how far awayobject 1202 is from vehicle 200, as discussed above in association withFIG. 12. When configured in this manner, measurements such as d₃-d₄ asshown in FIG. 12 may be measured and used for positional statusdetermination by lane positioning module 1320.

Distances from vehicle 200 to objects detected within the roadway, suchas object 1202, may be calculated in the manner described above inassociation with FIGS. 5-7. For example, system 100 may analyze imagedata, position data (e.g., GPS location information), map data, speeddata, and/or data from sensors included in vehicle 200 to calculate thedistances. System 100 may collect the data for analysis from, forexample, image acquisition unit 120, position sensor 130, and othersensors.

In some embodiments, processing unit 110 may construct the measurementsbased on estimation techniques using a series of time-based observationssuch as Kalman filters or linear quadratic estimation (LQE), and/orbased on available modeling data for different object types (e.g., cars,trucks, pedestrians, bicycles, road signs, etc.). The Kalman filters maybe based on a measurement of an object's scale, where the scalemeasurement is proportional to a time to collision (e.g., the amount oftime for vehicle 200 to reach the object).

Processing unit 110 may be configured in some embodiments to create a 3Dreconstruction of the visible portion of the roadway, using one or moremathematical algorithms or models as discussed previously. For example,as discussed above in association with FIG. 5, processing unit 110 maydetermine a look-ahead distance for vehicle 200, which may have a lowerbound ranging from 10 to 20 meters, may be calculated as the product ofthe speed of vehicle 200 and a determined look-ahead time. For example,as the speed of vehicle 200 decreases, the look-ahead distance may alsodecrease (e.g., until it reaches the lower bound). Additionally oralternatively, processing unit 110 may execute stereo image analysismodule 404 to perform stereo image analysis of the first and secondplurality of images to create a 3D map of the road in front of thevehicle and detect features within the images, such as object 1202.Processing unit 110 may estimate a position of object 1202 relative tovehicle 200 by observing differences that may exist relative to objectsappearing in at least two of the plurality of captured images within theimage data. For example, position, velocity, and/or accelerationrelative to vehicle 200 may be determined based on trajectories,positions, movement characteristics, etc. of features associated withobject 1202.

Object 1202 may be a relatively flat object, or it may be threedimensional. As discussed in further detail below, if the object is flator relatively flat (e.g., 2 cm or less, 1 cm or less, etc.), system 100may determine that no further action or change in course of vehicle 200is necessary. In the illustration of FIG. 14A, object 1202 is again acardboard box. In first image 1402, object 1202 has initial dimensionso_(x1), o_(y1), and o_(z1). System 100, via object detection module1310, may be configured to determine these dimensions based on thereference plane 1408, the detected lane constraints of the roadway, orother reference points of known size and/or location within the imagedata. In second image 1404, taken later, the object 1202 is now closerto the vehicle, and appears larger. Object 1202 now has seconddimensions o_(x2), o_(y2), and o_(z2). System 100 may determine thissecond set of dimensions in the same manner as the first.

In some embodiments, system 100 may use the determined dimensions ofobject 1202 in first image 1402 and at least a second image, such asimage 1404, to gather more information about the object and determineone or more attributes of the object. In some embodiments, system 100(via processing unit 110 and/or image processor 190) may be configuredto automatically recognize the object based on known objects previouslyprogrammed into an integrated database or a database accessible by theInternet. In these embodiments, system 100 may be configured to generateaudio or visual feedback to a human operator of vehicle 200.

System 100 may determine, for example, that one or more dimensionso_(x2), o_(y2), and o_(z2) from image 1404 are different from dimensionso_(x1), o_(y1), and o_(z1) from image 1402. If the second set ofdimensions from image 1404 are smaller than those taken from the firstimage 1402, the object is likely moving away from vehicle 200 and isgenerally not of concern. If the second set of dimensions is determinedto be larger than the first set of dimensions, then that determinationmay indicate that object 1202 is either stationary in the roadway ormoving towards vehicle 200, and additional actions may be taken. Forexample, system 100 may determine the height of object 1202(essentially, dimension o_(x1)/o_(x2)) relative to reference plane 1408to determine the approximate height of object 1202 relative to theroadway surface. In some embodiments, system 100 may be configured todetect objects as small as 2 cm tall. Based on the determined height ofthe object, system 100 may take one of several alternative actions, aswill now be discussed in further detail.

FIG. 14B provides an annotated view of the situation depicted in FIG.12. Vehicle 200 (here represented as vehicle 200 c) is once againtraveling in lane 810 of roadway 800. Vehicle 200 c is again equippedwith image capture devices 122 and 124; more or fewer devices may beassociated with any particular vehicle 200.

Object 1202 is located in lane 810 of roadway 800. Using the processdescribed above in association with FIG. 14A (and in further detailbelow in association with FIG. 15 and process 1500), system 100 may beconfigured to determine multiple attributes of object 1202 based on aplurality of images acquired from one or more of image capture devices122-126, and received via data interface 128. These attributes mayinclude the position and dimensions of object 1202, including its heightrelative to the surface of roadway 800.

As an initial step, system 100, via position sensor 130 and/or lanepositioning module 1320, may determine relative positions of vehicle 200c and object 1202. This determination may include determining laneconstraints or other such positional indicators associated with roadway800 and/or lanes 810/820. System 100 via processing unit 110 may thendetermine the position of vehicle 200 c within its current lane oftravel (here, lane 810) by calculating distances from the determinedconstraints. In the illustration of FIG. 14B, system 100 may for exampledetermine distances q₁ and q₂ from one or more surfaces of vehicle 200to the lane constraints. Further, system 100 may determine distance d₂from a surface of vehicle 200 c to a surface of object 1202. System 100may also determine relative positioning of object 1202 laterally withinthe lane, such as by measuring distance q₃ from a surface of the object1202 to the lane constraint separating lanes 810 and 820. Each of thesedetermined distances, such as q₁, q₂, etc., may be measured from anyportion of the interior or exterior of vehicle 200 c, including but notlimited to the front of vehicle 200 c, a portion of vehicle 200 c suchas a headlight or front license plate, a position as-installed of imagecapture devices 122 or 124, a determined centroid of vehicle 200 c, therear of vehicle 200 c, one or more windshields or mirrors associatedwith vehicle 200 c, wheels of vehicle 200 c, right or left sides orwindows of vehicle 200 c, a point associated with the roof of vehicle200 c, or a point associated with the chassis of vehicle 200 c.Distances d₂ and q₃ may also be measured to any portion inthree-dimensional space of object 1202.

Based at least on the dimensions of object 1202 determined as discussedabove in association with FIG. 14A, as well as the determined distancesillustrated in FIG. 14B, system 100 may determine one or more actions totake. System 100 may act via action module 1330 to send information andinstructions to various associated subsystems of vehicle 200 c to carryout the actions, including but not limited to throttling system 220,braking system 230, and steering system 240. The action taken by actionmodule 1330 may be further informed by a determination of the roadwayclearance height of vehicle 200 c. This clearance height may be a knownvalue stored in memory 140/150 of system 100, or it may be determinedanew in each driving situation based on various factors such as theinflation status of the tires, the road surface, weather conditions,etc. In some embodiments, the clearance height may be measured usingoptics and a reference point (e.g., from vehicle 200 c to a point on theroad surface) or by a sound echo, optical echo, radio frequency from theroad surface, etc. Action module 1330 may then determine one of severalalternative actions to undertake based on the determined height ofobject 1202 and the roadway clearance height of vehicle 200 c. Forexample, if the height of object 1202 is less than the roadway clearanceheight of vehicle 200 c, action module 1330 may determine that thesafest course of action is to simply do nothing, and allow vehicle 200 cto drive over object 1202 with the object oriented between the wheels ofvehicle 200 c. Alternatively, action module 1330 may determine that theheight of object 1202 is less than the roadway clearance height ofvehicle 200 c, but that a minor change in the directional course ofvehicle 200 c may permit vehicle 200 c to safely drive over object 1202.Further, system 100 and action module 1330 may determine that the heightof object 1202 exceeds the roadway clearance height of vehicle 200 c,and that corrective adjustments to the directional course of vehicle 200c may be required in order to avoid object 1202. These adjustments mayfurther require instructions to be sent to throttling system 220 and/orbraking system 230 in order to safely undertake the change in course.

This latter scenario is illustrated in FIG. 14B. This illustration is anexample only, and is not intended to be limiting. In the example of FIG.14B, system 100 has determined that the height of object 1202 is largerthan the roadway clearance height of vehicle 200 c. Accordingly, actionmodule 1330 may determine, based on detected lane constraints andmeasured distances as described above, that the best course of actionfor vehicle 200 c is to move slightly to the right to avoid object 1202.Action module 1330 may then send instructions to throttling system 220,braking system 230, and steering system 240 to effect this change indirectional course.

The projected location of vehicle 200 c as it passes object 1202 isrepresented by vehicle 200 d. System 100 may determine new andadditional distances in order to successfully complete the pass ofobject 1202. For example, system 100 may determine a distance q₄ asshown in FIG. 14B, corresponding to a safe distance needed from object1202 to a surface of vehicle 200 d. System 100 may further calculatedistances q₅ and q₆ via lane positioning module 1320, representing theposition of vehicle 200 d relative to detected lane constraints. Thesedistances may be sent to action module 1330 to assist in determining theproper directional course for vehicle 200 d. Ideally, vehicle 200 dwould successfully swerve around object 1202 while staying within theconstraints of lane 810. In some embodiments, however, it may benecessary for vehicle 200 d to move outside of lane 810's constraints inorder to avoid object 1202, or it may be necessary for vehicle 200 d tosimply change lanes, such as to lane 820. Once object 1202 is passed,action module 1330 may send an indication to throttling system 220,braking system 230, and steering system 240 that normal operation ofvehicle 200 may be resumed.

FIG. 15 illustrates a process 1500 for detecting an object in a roadway,consistent with disclosed embodiments. Steps of process 1500 may beperformed by one or more of processing unit 110, image acquisition unit120, position sensor 130, image processor 190, object detection module1310, lane positioning module 1320, or action module 1330.

At step 1510, process 1500 may include acquiring, using at least oneimage capture device 122, 124, and/or 126, a plurality of images of thearea surrounding vehicle 200. In some embodiments, the plurality ofimages may be of an area forward of vehicle 200. Processing unit 110 mayreceive the plurality of images from the image capture device(s) throughdata interface 128. The plurality of images may then be processed inreal time by image processor 190 of processing unit 110. Image processor190 may process the first image captured by the at least one imagecapture device, as well as at least one additional image captured at alater point in time. The at least one image capture device may beconfigured to have a field of view of between about 35 degrees and about55 degrees. Examples of acquired images and potential fields of view forsystem 100 are illustrated above in association with FIG. 14A.

At step 1520, process 1500 may determine a reference plane correspondingto the surface of roadway 800. The determination may be carried out byone or more of image processor 190, object detection module 1310, orlane positioning module 1320. The reference plane may be similar to theplane 1408 described above in association with FIG. 14A. The referenceplane may be determined based on the plurality of images captured by theat least one image capture device. In these embodiments, the referenceplane may be determined by analyzing the first image and at least asecond image from among the plurality of images. In some embodiments,the location of a roadway surface may be determined by analysis of theplurality of images by image processor 190.

The reference plane may be detected by way of parallax information, orby way of detecting one or more lane constraints on the roadway surface.In some embodiments, the reference plane may be determined by color ordarkness differences in the image data. Processing unit 110 may generateand propose multiple potential reference planes, with image processor190, and then select the best candidate. Once a reference plane isselected, information about its location may be stored within system100, such as in memory 140/150. In some embodiments, processing unit 110and/or image processor 190 may periodically redetermine the referenceplane based on changes in the roadway surface, such as elevation orcurvature changes, or when vehicle 200 turns onto a different roadway.In some embodiments, processing unit 110 may be configured to determinethe reference plane 1408 corresponding to the surface of roadway 800 asan average surface level associated with a road surface appearing in theplurality of images. This method may be useful, for example, whenroadway 800 is not perfectly flat or is otherwise unusual, such as ifthe road surface includes one or more of gravel, potholes, snow or ice,mud, vegetation, etc.

At step 1530, process 1500 may locate a target object in the first imageand at least one second image taken at a later time within the pluralityof images. Object detection module 1310 may locate the images based ontheir presence in multiple images among the plurality of images acquiredby the system, as illustrated above in association with FIG. 14A. Objectdetection module 1310 may determine one or more positional measurementsassociated with the detected object. These measurements, as discussedabove in association with FIGS. 14A-14B, may include relativemeasurements to other objects, such as vehicles or lane constraints. Thedetermination of object position may be assisted by a direct measuringtool, physical device, or module, such as position sensor 130. In otherembodiments, the determinations may include one or more visualdeterminations. The visual determinations may be directly determined bysoftware instructions executed by image acquisition unit 120 in themanner of a “range finder,” or may be determined from analysis of imagedata received from data interface 128 by image processor 190 andacquired by one or more of image capture devices 122-126. Oncedetermined, the object's position may be stored within system 100, suchas in memory 140/150, and may be periodically updated by objectdetection module 1310.

At step 1540, process 1500 may determine one or more attributes of thedetected object(s). In some embodiments, the object may be recognized asa particular object, based on information stored in a databaseassociated with system 100. As discussed above, these determinedattributes may include dimensions of the detected object, such as itsheight relative to the detected reference plane. System 100 may beconfigured to determine the height of any object within a roadway thatextends at least about 2 cm above the reference plane. In alternativeembodiments, system 100 may be configured to detect and determineattributes for any object extending between 2 and 10 cm above thereference plane. In other embodiments, the determination of targetobject height may also be based on a detected velocity of the vehicle.The velocity may be determined based on information received from one ormore of throttling system 220 or braking system 230. Additional detailson the calculation of a reference plane for the road surface andcalculating the height of a target object may be found in U.S. patentapplication Ser. No. 14/554,500, filed Nov. 26, 2014, which is herebyincorporated by reference in its entirety.

System 100 may be configured to determine actions associated with thedetected object based on the determined attributes. For example, system100 and action module 1330 may be configured to take no substantiveaction unless the determined height of the detected object exceeds athreshold level. In some embodiments, the threshold level may be about 2cm, as discussed above, or may be higher (e.g., 3 cm, 4 cm, 5 cm, 6 cm,etc.) depending on user or system preferences. For example, in someembodiments, the threshold may be associated with a detected roadwayclearance height of vehicle 200. If processing unit 110 determines fromthe acquired information that height of the object does not exceed theconfigured threshold (Step 1550: NO), then process 1500 may revert backto the beginning, and system 100 may continue to monitor the status ofpotential objects within a roadway.

Alternatively, if system 100 determines that an action may be needed inresponse to the presence of a detected object in the roadway (Step 1550:YES), then process 1100 may proceed to step 1560. At step 1560, process1500 may cause vehicle 200 to change directional course to evade orotherwise pass a detected object. Via action module 1330, processingunit 110 may send an instruction to one or more systems associated withvehicle 200. For example, an instruction may be sent to steering system240, instructing vehicle 200 to merge left or right into the next lane.Alternatively, instructions may be sent to swerve around the object, orto change directional course such that the vehicle simply drives overthe object with the object situated between the wheels. Additionalinstructions may be sent; for example, image processor 190 may determinefrom real-time image data that traffic or other obstacles prevent animmediate change in directional course. In these embodiments, aninstruction may first be sent to braking system 230 and/or the turnsignals, and vehicle 200 may decelerate or be brought to a stop untilthe directional course change is possible. At that time, the instructionto steering system 220 may be safely made and the vehicle may turn tochange course.

In some alternative embodiments, where vehicle 200 is partially or fullyunder the control of a human operator, step 1560 may instead comprisegenerating feedback to the operator informing that an object is detectedin the roadway and that a change in directional course is suggested orrequired. In some embodiments, an automated system such as system 100and its action module 1330 may take over operation of the car and causethe vehicle to change course automatically as described above if thehuman operator does not respond to the feedback within a certain timeperiod. The time period may be predetermined, or may be calculated basedon visual determinations such as those described above in associationwith FIGS. 12 and 14A-14B.

Ultra-Wide Traffic Light Detection System

In some embodiments, system 100 may identify objects located outside ofa sightline of a typical driver location in vehicle 200. The typicaldriver location in vehicle 200 is on the left side of the front seat incountries that drive on the right-hand side of the road, and on theright side of the front seat in countries that drive on the left-handside of the road. The U.S. National Highway Traffic SafetyAdministration recommends that drivers place themselves 10 inches (25.4centimeters) from the driver air bag for safety reasons. This distanceis measured from the center of the steering wheel to the driver'sbreastbone. Because vehicle 200 is not transparent, portions of vehicle200 (e.g., the roof, doors, windshield frame, etc.) may obscure theangle-of-view from the typical driver location. In addition, whilehumans have an almost 180-degree forward-facing horizontal field ofview, the vertical range of the field of view in humans is significantlyless. However, a driver assistance system may use an image capturedevice (e.g., image capture device 122) that has an extendedangle-of-view that exceeds the angle-of-view of a driver who is seatedin the typical driver location. The extended angle-of-view may enablesystem 100 to identify objects in the environment of vehicle 200 thatare otherwise undetectable by the driver. Moreover, system 100 maydetermine a change associated with the identified objects, and actaccordingly. For example, system 100 may determine a change in status ofa traffic light located in a region above vehicle 200. As anotherexample, system 100 may determine the trajectory of different objects(e.g., pedestrians) located in a region beside vehicle 200.

FIGS. 16 and 17 are diagrammatic representations of vehicle 200 in twosituations, consistent with disclosed embodiments. As illustrated inFIGS. 16 and 17, vehicle 200 may include image capture device 122associated with a first angle-of-view. The first angle-of-view isvertically represented by FOV 1610, as shown in FIG. 16, andhorizontally represented by FOV 1710, as shown in FIG. 17. Alsoillustrated in FIGS. 16 and 17 is a second angle-of-view from thetypical driver location in vehicle 200. The second angle-of-view isvertically represented by FOV 1620, as shown in FIG. 16, andhorizontally represented by FOV 1720, as shown in FIG. 17. Consistentwith embodiments of this disclosure, the first angle-of-view associatedwith image capture device 122 may be broader than the secondangle-of-view associated with the typical driver location in vehicle200. The delta between the broader first angle-of-view and the secondangle-of-view is referenced in this disclosure as an extended zone. Insome embodiments, the extended zone may include objects located outsidethe sightline of the typical driver location in vehicle 200, whichsystem 100 may identify. In addition, system 100 may trigger one or moreactions (e.g., cause navigational responses) based on a determinedchange associated with the objects identified in the extended zone.

FIG. 16 illustrates the first example for determining a changeassociated with one or more objects located outside of the sightline ofthe typical driver location in vehicle 200. The angular range of FOV1610 and FOV 1620 is used in FIG. 16 for illustrative purposes. In someembodiments, FOV 1620 may be, for example, about 65 degrees, and FOV1610 may be, for example, higher than 65 degrees. In other embodiments,FOV 1620 may be, for example, about 135 degrees, and FOV 1610 may be,for example, higher than 150 degrees, higher than 160 degrees, or higherthan 170 degrees. In some embodiments, FOV 1610 and FOV 1620 may bemeasured in an upward direction from the horizon. As shown in FIG. 16,the extended zone may include a traffic lamp fixture 1630 located in aregion above vehicle 200. Traffic lamp fixture 1630, which is locatedoutside of the sightline of the typical driver location, displays a redlight. In some embodiments, system 100 may be able to detect trafficlights and changes in the displayed color of traffic lights insituations where the driver of vehicle 200 cannot even see the trafficlight fixture.

One example of these situations is when a driver of vehicle 200 appliesthe brakes in a manner that causes vehicle 200 to stop over the stoppingline of the intersection and/or positions vehicle 200 too close totraffic light fixture 1360. Another example of these situations is whenvehicle 200 reaches the stopping line at an intersection, and trafficlamp fixture 1630 is placed on a high pole so it can be seen from adistance. Because traffic lamp fixture 1630 is located outside FOV 1620,when the light in traffic lamp fixture 1630 changes to green, a driverwho sits in the typical driver location would not be able to detect thestatus change. However, system 100 would be able to detect the statuschange of the traffic light because traffic lamp fixture 1630 is locatedwithin FOV 1610, and thus in view of image capture device 122.Identifying traffic lights located in a region above vehicle 200 mayimprove safety and driver convenience. For example, when considering thesafety of the driver and the passengers of vehicle 200, if traffic lampfixture 1630 is not inside FOV 1620, it may be unsafe to back up vehicle200 until traffic lamp fixture 1630 is within FOV 1620. In addition,even if backing up vehicle 200 is not dangerous, it might beinconvenient for the driver to back up vehicle 200. In one embodiment,after determining that the light in traffic lamp fixture 1630 has turnedto green, system 100 may cause vehicle 200 to accelerate to cross theintersection.

FIG. 17 illustrates the second example for determining a changeassociated with the objects located outside of the sightline of thetypical driver location in vehicle 200. The angular range of FOV 1710and FOV 1720 is used in FIG. 17 for illustrative purposes. In someembodiments, FOV 1720 may be, for example, about 180 degrees, and FOV1710 may be, for example, higher than 200 degrees, higher than 240degrees, or higher than 300 degrees. As shown in FIG. 17, the extendedzone may include a pedestrian 1730 located in a region beside vehicle200. Pedestrian 1730 may be moving in a certain direction. In somecases, pedestrian 1730 may be located at the edge of FOV 1720, butbecause the driver is focused on the road ahead, he or she may fail todetect pedestrian 1730. In contrast, system 100 not only has a broaderFOV, it does not have to focus on the road ahead and can identifyobjects located in any region of its FOV. Because pedestrian 213 islocated within FOV 1710, system 100 may detect pedestrian 1730 anddetermine the pedestrian's walking direction trajectory. In otherembodiments, system 100 may identify an object located beside vehicle200, determine the object's moving direction trajectory, determinewhether the object is on a possible collision course with vehicle 200,and act accordingly. For example, after determining that pedestrian 1730is on a possible collision course with vehicle 200, system 100 maycontrol the vehicle's horn to generate a type of audio alarm.

As discussed above, FIGS. 3B-3D illustrate exemplary camera mount 370,which is configured to be positioned behind rearview mirror 310 andagainst a vehicle windshield. As shown in FIGS. 3B-3D, camera mount 370may include a plurality of image capture devices (122, 124, and 126). Inone embodiment, the combined angle-of-view of image capture devices 122,124, and 126 forms the first angle-of-view. For example, in order forFOV 1710 to be higher than 300 degrees, a combination of six imagecapture devices may be used. In a different embodiment, at least one ofthe image capture devices depicted in FIGS. 3B-3D (e.g., image capturedevice 122) includes an ultra-wide lens. The ultra-wide lens may enableimage capture device 122 to have the first angle-of-view. In someembodiments, the ultra-wide lens may have a focal length shorter thanthe short side of the image sensor of image capture device 122.Alternatively, the ultra-wide lens may have a focal length shorter thanthe long side of the image sensor, but longer then the short side of theimage sensor. Consistent with an embodiment of the present disclosure,image capture device 122 may be configured to capture images of trafficlights located in a conic section zone of, for example, between 150 to180 degrees, 150 to 190 degrees, 160 to 180 degrees, or 170 to 180degrees relative to a location of image capture device 122.

FIG. 18 is an exemplary block diagram of a memory configured to storeinstructions for performing one or more operations, consistent with thedisclosed embodiments. As shown in FIG. 18, memory 140 or 150 may storean object identifying module 1802, a status determining module 1804, anda system response module 1806. The disclosed embodiments are not limitedto any particular configuration of memory 140 or 150. Further,processing unit 110 may execute the instructions stored in any ofmodules 1802-1806 included in memory 140 or 150.

In one embodiment, object identifying module 1802 may store softwareinstructions (such as computer vision software) which, when executed byprocessing unit 110, identifies an object located outside of thesightline of the typical driver location in vehicle 200. In oneembodiment, processing unit 110 may scan a plurality of images, compareportions of the images to one or more predetermined patterns, andidentify a traffic light located in a region above vehicle 200. Inanother embodiment, processing unit 110 may scan a plurality of images,compare portions of the images to one or more predetermined patterns,and identify an object located in a region beside vehicle 200. In someembodiments, module 1802 may store software instructions, which, whenexecuted by processing unit 110, identifies intersections and determinesthat a traffic light may be nearby. One way for processing unit 110 toidentify an intersection is by using historical image data. For example,processing unit 110 may determine that a traffic light may be nearbyeven when no traffic light is visible in a current image, but thetraffic light was visible in a prior image (e.g., a prior frame, fiveframes ago, etc.). Additionally or alternatively, processing unit 110may identify in image data objects indicative of a close intersection.For example, processing unit 110 may determine that a traffic light maybe nearby by identifying cross roads in image data of the road ahead. Inanother example, processing unit 110 may identify in image data the backof other traffic lights regulating traffic going in the oppositedirection to the driving direction of vehicle 200. Additionally oralternatively, processing unit 110 may use map data (such as data frommap database 160) to identify an intersection. For example, processingunit 110 may consider general road shape descriptions extracted from mapdatabase 160 to determine that a traffic light may be nearby.

In another embodiment, module 1802 may store software instructions,which, when executed by processing unit 110, corrects optical distortionin the plurality of images to enable identifying the objects. Theoptical distortion may be a result of image capture device 122 using anultra-wide lens. For example, the optical distortion may be a barreldistortion in which the image magnification decreases with distance fromthe optical axis, and thus the distortion may be most significant closeto the edge of the image frame. Consistent with the present disclosure,a plurality of algorithms may be used to correct the optical distortionthat may result from using an ultra-wide lens. For example, an algorithmmay be used to correct the optical distortion to identify an objectclose to the edge of the image frame. As another example, a differentalgorithm may be used to identify the status of an object located at theedge of the image frame. In another embodiment, the image capture devicemay include a plurality of image capture devices (e.g., image capturedevices 122, 124, and 126), and each image capture device may beassociated with a different field-of-view. Module 1802 may storesoftware instructions, which, when executed by processing unit 110,correctly combines images retrieved from the plurality of image capturedevices. For example, even when there may be some overlap in thefield-of-view of two image capture devices, different parts of an objectmay be captured by the different image capture devices. Accordingly,object identifying module 1802 may include instructions for identifyingobjects in regions that span adjoining images captured by differentimage capture devices.

In one embodiment, status determining module 1804 may store softwareinstructions which, when executed by processing unit 110, determines astatus of an object identified in the image data, and located at leastin part within an extended zone. Consistent with the present disclosure,the extended zone includes the region between the field-of-view of animage capture device (e.g., FOV 1610 and 1710) and the field-of-view ofa typical driver location in vehicle 200 (e.g., FOV 1620 and 1720). Inone embodiment, status determining module 1804 may store softwareinstructions, which, when executed by processing unit 110, determinesthe status of a traffic light located in a region above vehicle 200. Forexample, processing unit 110 may determine the status of a traffic lightby comparing one or more images to stored patterns. In addition, achange in the status of the traffic light may be determined based on thedifferences from between the stored patterns and two or more imagestaken at different times. In another embodiment, status determiningmodule 1804 may store software instructions, which, when executed byprocessing unit 110, determines the trajectory of moving objects locatedin a region beside vehicle 200. For example, processing unit 110 maydetermine the trajectory of the moving objects by comparing a pluralityof images to each other. In addition, processing unit 110 may determinewhether the moving object is on a possible collision course with vehicle200.

In one embodiment, system response module 1806 may store softwareinstructions which, when executed by processing unit 110, causes one ormore responses in vehicle 200. One type of system response may includenavigational responses, as described above in detail with reference tonavigational response module 408. Other types of system responses mayinvolve control throttling system 220, braking system 230, and/orsteering system 240. For example, processing unit 110 may transmitelectronic signals that cause system 100 to physically depress the brakeby a predetermined amount or ease partially off the accelerator ofvehicle 200. Further, processing unit 110 may transmit electronicsignals that cause system 100 to steer vehicle 200 in a particulardirection. Such responses may be based on the determination of therelevant traffic light. Additional system responses may includeproviding various notifications (e.g., warnings and/or alerts). Thenotifications may include reporting the status of the traffic lightlocated above vehicle 200, reporting a change in status of the trafficlight located above vehicle 200, reporting an existence of an objectbeside a vehicle, and reporting that the object is on a collision coursewith vehicle 200.

FIG. 19 is a flowchart showing an exemplary process 1900 for navigatinga vehicle based on a change in status of a traffic light located in aregion above vehicle 200, as depicted in FIG. 16. A similar process iscontemplated for controlling a vehicle based on the trajectory of movingobjects located in a region beside vehicle 200, as depicted in FIG. 17.Accordingly, exemplary process 1900 may be applied to a variety ofsituations, including the situation depicted in FIG. 17.

As shown in FIG. 19, at step 1902, image capture device 122 may acquirea plurality of images of a traffic light located in a region abovevehicle 200, outside of the sightline of a typical driver location. Atstep 1904, processing unit 110 may determine the relevancy of thetraffic light captured in the plurality of images based on at least oneindicator of the vehicle position, for example, lane markers. In someembodiments, this step may be optional and omitted. At step 1906,processing unit 110 may determine the status of the traffic light basedon an analysis of at least a first image from among the plurality ofimages. At step 1908, processing unit 110 may determine a change in thestatus of the traffic light based on at least a second image from amongthe plurality of images. For example, the change in the status of thetraffic light may be from a red light to a green light. Afterdetermining that the status of a traffic light has changed, at step1910, processing unit 110 may cause a system response based on thedetermined change in status. These steps of process 1900 are discussedin further detail below.

At step 1902, image capture device 122 may acquire a plurality of imagesof a traffic light located in a region above vehicle 200. Consistentwith disclosed embodiments, image capture device 122 may include one ormore image capture devices. Consistent with some embodiments, thetraffic light may be located outside of a sightline of a typical driverlocation in vehicle 200. For example, the traffic light may be locatedin the extended zone as described with reference to FIG. 16. In someembodiments, image capture device 122 may be located within vehicle 200.In addition, although in some embodiments image capture device 122 maybe located behind a rearview mirror and against a vehicle windshield,processing unit 110 may determine the status of the traffic lightthrough the windshield.

At step 1904, processing unit 110 may determine the relevancy of thetraffic light located in the region above vehicle 200 based on at leastone indicator of vehicle position. A detailed disclosure of how system100 may determine the relevancy of traffic lights is provided below inconnection with FIGS. 20-23. In some embodiments, the indicator ofvehicle position may be derived from analyzing the plurality of images.For example, processing unit 110 may determine the relative position ofthe traffic light to vehicle 200, based on identification of a lanemarker in the plurality of images. Additionally or alternatively, theindicator of vehicle position may be derived from geographic positiondata retrieved from position sensor 130. For example, processing unit110 may determine the relative position of the traffic light to vehicle200, based on GPS signals or local positioning signals. In someembodiments, processing unit 110 may compare the GPS acquired vehiclelocation to map data to determine the relevance of the traffic light.For example, processing unit 110 may conduct an analysis usinginformation derived from image data and the map data. Using the derivedinformation, processing unit 110 may determine a correspondence betweenthe detected traffic light located in the region above vehicle 200 andthe lane vehicle 200 is currently driving.

At step 1906, processing unit 110 may determine the status of thetraffic light based on an analysis of at least a first image from amongthe plurality of images. In some embodiments, the status of the trafficlight located in the region above vehicle 200 is associated with thecolor the traffic light indicates. For example, a steady green light mayindicate that vehicle 200 can continue after yielding to other vehicles,bicycles, or pedestrians in the road; a flashing yellow light is awarning and may indicate that vehicle 200 should proceed with caution;and a steady red light may indicate that vehicle 200 needs to stop.Additionally or alternatively, the status of the traffic light may alsoinclude additional information associated with the traffic light. Theadditional information may be the type of the traffic light. Forexample, the traffic light located in the region above vehicle 200 maybe a pedestrian crossing signal. Accordingly, processing unit 110 maydetermine that pedestrians may cross the road in front of vehicle 200.In some embodiments, in order to determine the status of the trafficlight located in the region above vehicle 200, processing device 110 maycorrect optical distortion in the at least a first image. The opticaldistortion may occur when image capture device 122 includes anultra-wide lens.

At step 1908, processing unit 110 may determine a change in status ofthe traffic light based on at least a second image from among theplurality of images. In some embodiments, the status of the trafficlight located in the region above vehicle 200 may change, for example,from a red light to a green light. In one embodiment, the at leastsecond image may include two or more images subsequent to the at least afirst image. In addition, processing unit 110 may process only a portionof the at least a second image. For example, when vehicle 200 is notmoving the position of the traffic light in the frame remains the same.Accordingly, in some embodiments, upon identifying the portion of theframe that includes the traffic lamp fixture in the at least a firstimage, processing unit 110 may determine the change in status byprocessing only the same portion of the frame in the at least a secondimage. For example, in some embodiments, by processing only the portionof the frame that includes the traffic lamp fixture, processing unit 110may save processing power.

At step 1910, processing unit 110 may cause a system response based onthe determined change in status. In some embodiments, processing unit110 may cause one or more system responses (e.g., two or moreresponses), including responses of different types. One type of systemresponse may include providing various notifications (e.g., warningsand/or alerts) to the driver of vehicle 200. The notifications may beprovided via speakers 360 or via a touch screen 320. For example, thesystem response may include reporting the change in status to the driverof vehicle 200. Another type of system response may include navigationalresponses, the navigational responses may include, for example,accelerating to a driver-specified cruise speed.

Identifying Relevant Traffic Lights

In some embodiments, system 100 may distinguish between relevant andirrelevant (or less relevant) traffic lights. During a typical drivingsession, vehicle 200 may cross one or more junctions, such asintersections, having multiple traffic lights. For example, one or moreof the traffic lights at an intersection may regulate traffic travelingtoward the intersection in a particular direction. Accordingly, one ormore traffic lights may regulate whether vehicle 200 may continue totravel through the intersection or whether vehicle 200 must stop at theintersection. However, in addition to the traffic lights that regulatethe lane in which vehicle 200 is traveling, other traffic lights thatregulate traffic in other lanes may be visible to vehicle 200.Navigating vehicle 200 according to any of the traffic lights thatregulate lanes other than the lane in which vehicle 200 is traveling mayresult in navigational responses undesirable or inapplicable to theintended route of vehicle 20. Accordingly, to enable autonomous controlof vehicle 200 in a manner appropriate to the intended navigational pathof vehicle 200, system 100 may identify which of a plurality of trafficlights is regulating traffic in the lane in which vehicle 200 istraveling while disregarding (or placing less emphasis on) other trafficlights that regulate other lanes of traffic. Further, after system 100identifies a relevant traffic light, system 100 may identify a status ofthe traffic light (e.g., red, yellow, green) and implement anappropriate navigational response. For example, system 100 maydiscontinue cruise control and apply the brakes when a red light isrecognized that regulates the lane in which vehicle 200 is traveling orwhen a yellow light is recognized that regulates the lane in whichvehicle 200 is traveling and vehicle 200 is beyond a predetermineddistance of a junction.

Distinguishing between relevant and irrelevant (or less relevant)traffic lights on a road may be complex. FIG. 20 and FIG. 21 illustrateexamples of situations where driving based on information derived froman inapplicable traffic light may be undesirable. In FIG. 20, vehicle200 a is traveling on a multilane road. Each lane of the road isassociated with a different traffic lamp fixture. Vehicle 200 a isapproaching an intersection and is traveling in a lane designated forproceeding through the intersection and to the other side of theintersection. Also shown in FIG. 20, vehicle 200 b is traveling in alane that allows traffic to continue to travel straight and through theintersection or that allows traffic to make a right turn. The trafficlamp fixture associated with the lane in which vehicle 200 b istraveling includes a right-turn traffic light. As another example, inFIG. 21, vehicle 200 a reaches a junction of non-perpendicular roads.Multiple traffic lamp fixtures (e.g., traffic lamp fixtures 2122 and2124), including some that do not regulate the lane in which vehicle 200a is traveling, may be visible to vehicle 200 a due to the orientationof the junction.

Returning to FIG. 20, an intersection 2000 is shown that has thefollowing driving options: road A has lanes leading to roads B, D, andF; road C has lanes leading to roads D and F; road E has lanes leadingto roads F, H, and B; and road G has lanes leading to roads H, B, and D.Road A is associated with four traffic lamp fixtures 2012, 2014, 2016,and 2018. In the situation shown, each traffic lamp fixture regulates adifferent lane. Road C is associated with two traffic lamp fixtures 2022and 2024. Traffic lamp fixture 2024 includes a traffic light 2024 a forcontinuing straight and a traffic light 2024 b for right turns (e.g.,displaying a green arrow when a right turn is authorized). Road E isassociated with three traffic lamp fixtures 2032, 2034, and 2036. Androad G is associated with two traffic lamp fixtures 2042 and 2044.Traffic lamp fixture 2042 includes a traffic light 2042 a for continuingstraight and a traffic light 2042 b for left turns. In the situationillustrated in FIG. 20, traffic light 2024 b and traffic lamp fixtures2012, 2018, and 2032 display green lights while all the other trafficlamp fixtures display red lights. Of course, many other road variationsand relative traffic light configurations may exist in addition to thoseshown in FIG. 20.

In the situation illustrated in FIG. 20, vehicle 200 a is located onroad A in a lane that continues straight through intersection 2000 toroad D. However, the field-of-view of an image capture device includedin vehicle 200 a may include both traffic lamp fixtures 2012 and trafficlamp fixture 2014 (or even additional fixtures). Vehicle 200 a arrivedat intersection 2000 when both traffic lamp fixtures 2012 and 2014displayed red lights, and only recently the light in traffic lampfixture 2012 has turned green. In this situation, it is important thatthe traffic light detection system 100 of vehicle 200 a recognizes thatthe green light of fixture 2012 is not applicable to vehicle 200 a.Rather, system 100 should base any determined navigational response onthe status of the lamp associated with the more relevant fixture 2014.

In another aspect of the situation illustrated in FIG. 20, vehicle 200 bis located on road C in a lane that continues straight throughintersection 2000 to road F, and that lane also allows traffic to make aright turn to road D. Vehicle 200 b faces traffic lamp fixture 2024 thatincludes a traffic light 2024 a for continuing straight and a trafficlight 2024 b for right turns. Vehicle 200 b arrived at intersection 2000when traffic lamp fixture 2024 displayed red lights for continuingstraight (traffic light 2024 a) and for right turns (traffic light 2024b), and only recently traffic light 2024 b has turned green. This meansthat the current status of traffic lamp fixture 2024 prohibits trafficfrom driving straight and turning right. An undesirable situation mightoccur if vehicle 200 b acts on the status of light 2024 b withoutrecognizing and accounting for the status of light 2024 a. For example,if vehicle 200 b drives to road F (i.e., drives straight) based on theinformation of traffic light 2024 b (showing a green light), vehiclesdriving from road E to road B may create a hazard for vehicle 200 b.

Turning to FIG. 21, FIG. 21 illustrates an intersection 2100 withnon-perpendicular roads that has the following driving options: road Ahas two lanes, one lane leading to road C and the other lane leading toroad E; road B has two lanes leading to road C, and one lane that canalso lead to road E; and road D has a single lane leading to road E.Road A is associated with two traffic lamp fixtures 2112 and 211, andeach traffic lamp fixture regulates a different lane. Road B is alsoassociated with two traffic lamp fixtures 2122 and 2124, and eachregulates a different lane. Road D is associated with a single trafficlamp fixture 2132. In the situation illustrated in FIG. 21, traffic lampfixture 2122 displays a green light, while all the other traffic lampfixtures display red lights. As shown in FIG. 21, vehicle 200 a islocated on road A in a lane that continues straight through intersection2000 to road C. However, the field-of-view of image capture device 122included in vehicle 200 a may include both traffic lamp fixture 2114 andtraffic lamp fixture 2122. In this example, vehicle 200 a has arrived atintersection 2100 when both traffic lamp fixtures 2114 and 2122 werered, and only recently the light in traffic lamp fixture 2122 has turnedgreen. In such a situation, it may be important that vehicle 200 adetermines its navigational response based on the status of trafficfixture 2114, rather than on the status of traffic fixture 2122.

The three situations depicted in FIGS. 20 and 21 provide just a fewexamples of road situations in which it may be helpful to have a driverassistance system that can distinguish between relevant and irrelevanttraffic lamp fixtures, to determine the status of the traffic lightsincluded in the relevant traffic lamp fixture(s), and to generate andtake an appropriate navigational response based on the status. Further,these examples demonstrate that the system may need to evaluate multipletraffic lamp fixtures that may face a vehicle in order to identify thetraffic lamp fixture that is most applicable to the lane in which thevehicle is traveling or to the intended travel direction of the vehicle(especially where, for example, multiple traffic lights or traffic lampfixtures may be associated with a single lane in which the vehicle istraveling).

As discussed above, system 100 may distinguish between relevant andirrelevant traffic lamp fixtures, determine the status of the trafficlights included in the relevant traffic lamp fixture(s), and generateand take an appropriate navigational response based on the status invarious driving scenarios. For example, as vehicle 200 approaches anintersection, system 100 may determine which traffic light is relevant,determine a status of that traffic light, and find any other relevantinformation in images captured by one or more of image capture devices122-126. If the traffic light is red, system 100 may cause vehicle 200to apply its brakes. If the traffic light is green, system 100 may causevehicle 200 to continue. If the traffic light is yellow, system 100 maydetermine a distance to the intersection and/or an estimate time to theintersection based on analysis of images, the speed of vehicle 200,and/or positional data (e.g., GPS data). If vehicle 200 is within apredetermined time (e.g., five seconds, ten seconds, etc.) and/ordistance (e.g., one meter, five meters, ten meters, etc.) threshold,system 100 may cause vehicle 200 to continue. If vehicle 200 is notwithin the predetermined time threshold and/or distance threshold,system 100 may cause vehicle 200 to stop before reaching theintersection. As another example, when vehicle 200 is stopped at atraffic light, after the traffic light changes its status from red togreen, system 100 may cause a navigational response that includesapplying the accelerator, releasing the brakes, and steering through anintersection, for example.

FIG. 22 is an exemplary block diagram of a memory 140/150 configured tostore instructions for performing one or more operations, consistentwith the disclosed embodiments. Memory 140/150 may be accessed byprocessing unit 110. As discussed above, processing unit 110 may includevarious devices, such as a controller, an image preprocessor, a centralprocessing unit (CPU), support circuits, digital signal processors,integrated circuits, memory, or any other types of devices for imageprocessing and analysis. Accordingly, memory 140/150 may be accessed byprocessing unit 110, may be integrated with processing unit 110, or manyincluded in an embedded system together with processing unit 110. Asshown in FIG. 22, memory 140 or 150 may store a relevancy determiningmodule 2202, a status determining module 2204, and a system responsemodule 2206. The disclosed embodiments are not limited to any particularconfiguration of memory 140. Further, processing unit 110 may executethe instructions stored in any of modules 2202-2206 included in memory140 or 150.

In one embodiment, relevancy determining module 2202 may store softwareinstructions (such as computer vision software) which, when executed byprocessing unit 110, determines a relevance of each of the plurality oftraffic lamp fixtures to vehicle 200. For purposes of this disclosure,determining the relevancy of traffic lamp fixtures may include executingone or more assessments. In one embodiment, processing unit 110 mayassess the orientation of each of a plurality of traffic lamp fixtureswith respect to the vehicle. For example, in the situation illustratedin FIG. 21 the field-of-view of image capture device 122 includes bothtraffic lamp fixture 2114 and traffic lamp fixture 2122. However, theorientation of traffic lamp fixture 2114 suggests that traffic lampfixture 2114 is more relevant to vehicle 200 a than fixture 2122.

Other assessments may be applicable to the decision making process. Forexample, processing unit 110 may assess the distance of each of theplurality of traffic lamp fixtures with respect to the vehicle.Additionally or alternatively, processing unit 110 may assess whichtraffic lamp is facing a front portion of vehicle 200. Additionally oralternatively, processing unit 110 may use the identified lane markersto divide the area forward of vehicle 200 to a plurality of zones,associate each identified traffic lamp fixture with a zone, and assesswhich zone is the most relevant for vehicle 200. Additionally oralternatively, processing unit 110 may compare a vehicle location thatwas acquired via GPS to map data to determine the relevance of thetraffic light. For example, vehicle 200 may access map data thatincludes information about the possible driving options at a number oflocations. By using the GPS acquired vehicle location, processing unit110 may determine which driving options are available to vehicle 200approaching a junction, and use this information to determine therelevance of the traffic lights at the junction to vehicle 200.

In one embodiment, status determining module 2204 may store softwareinstructions which, when executed by processing unit 110, determine astatus of a traffic light included in at least one traffic lampdetermined to be relevant to the vehicle. As described above, the statusof the traffic light may be also associated with the information thetraffic light provides. In a typical traffic light, the color beingdisplayed and the relative location of the illuminated traffic light mayprovide basic information relevant to vehicle 200.

Status determining module 2204 may derive additional information fromthe environment of the relevant traffic lamp fixture and fromnon-typical traffic light included in the relevant traffic lamp fixture.For example, the relevant traffic lamp fixture may have in its proximitya sign that includes relevant text, e.g., a sign stating specific daysand hours. Accordingly, in some embodiments, status determining module2204 may derive information from the text included in signs associatedwith the relevant traffic fixture. For example, status determiningmodule 2204 may implement optical character recognition techniques torecognize text in the signs. Status determining module 2204 may thencompare the recognized text to a database to determine the informationprovided by the sign. As another example, the relevant traffic lampfixture may include a pedestrian crossing signal. Status determiningmodule 2204 may determine that the status of the pedestrian crossingsignal means that pedestrians may cross the road in front of vehicle200, while vehicle 200 has a green light to turn right.

In one embodiment, system response module 2206 may store softwareinstructions (such as computer vision software) which, when executed byprocessing unit 110, causes one or more responses in vehicle 200. Onetype of system response may include one or more navigational responses,as described above in detail with reference to navigational responsemodule 408. Other types of system responses may provide control signalsto throttling system 220, braking system 230, and/or steering system240. For example, processing unit 110 may transmit electronic signalsthat cause system 100 to physically apply the brakes by a predeterminedamount or ease partially off the accelerator of vehicle 200. Further,processing unit 110 may transmit electronic signals that cause system100 to steer vehicle 200 in a particular direction. Such responses maybe based on the determination of the relevant traffic light. Further, ifpedestrians may cross the road in front of vehicle 200, then the systemresponses may include executing additional processing of the image data.For example, processing unit 110 may confirm that a pedestrian in theproximity of vehicle 200 is not walking in a trajectory that would placethe pedestrian in danger while vehicle 200 is moving.

The location of the traffic lamp fixtures in a junction can be before,after, or in the middle of the junction. Identifying the position ofeach traffic light fixture in the junction may be used, for example, todetermine the relevancy of the traffic lamp fixtures. In someembodiments, processing unit 110 may estimate the distance of one ormore traffic lamp fixtures with respect to vehicle 200 to create a 3Dmodel of a junction. In one embodiment, the 3D model of the junction maybe stored for future usage. The 3D model of the junction may include oneor more of the following: a 3D position for one or more traffic lightfixtures, the relative distance between each traffic light fixture andother traffic light fixtures in the junction, the direction(s) eachtraffic light fixture refers to, and the relative distance between eachtraffic light fixture and the stop line of the junction. In addition,the 3D model may be periodically updated using details recognizable whenvehicle 200 approaches the junction. Examples of the recognizabledetails may include arrows in traffic lights, the lane marking in thejunction, etc. In one example, when time vehicle 200 passes thejunction, recognized details are compared to the information stored inthe 3D model, and if appropriate, the 3D model is updated.

Consistent with embodiments of the present disclosure, the 3D model maybe used to determine the relevancy of one or more traffic lamp fixtures.In order to activate braking system 230 with sufficient time to stopvehicle 200 before the junction, system 100 may determine which trafficlight is relevant to vehicle 200 at a distance of about eighty metersfrom the junction. As vehicle 200 approaches a junction, informationderived from the captured images may be compared to the 3D model to finda match. For example, processing unit 110 may compare the relativedistance between recognized traffic light fixtures in the junction tothe 3D model to determine the relevancy of one or more traffic lightfixtures. Using the 3D model, processing unit 110 may identify arelevant traffic light when vehicle 200 is more than 50 meters, 75meters, 100 meters, or 125 meters from the junction. As another example,processing unit 110 may compare the relative distance between recognizedtraffic light fixtures in the junction to the 3D model to determine thedistance to the junction's stop line, even when the stop line is notvisible from the current location of vehicle 200. Using the 3D model,processing unit 110 may determine the distance to the junction's stopline when vehicle 200 is more than 50 meters, 75 meters, 100 meters, or125 meters from the junction. Further details regarding relevancydetermining module 2202, a status determining module 2204, and a systemresponse module 2206 are provided below in connection with thediscussion of FIG. 23.

FIG. 23 is a flowchart showing an exemplary process 2300 for identifyingrelevant traffic lights, consistent with disclosed embodiments. System100 may implement process 2300 to address the situations depicted in,for example, FIGS. 20 and 21.

As illustrated in FIG. 23, at step 2310, image capture device 122 mayacquire images of an area forward of vehicle 200. The area may include aplurality of traffic lamp fixtures, each including at least one trafficlight. At step 2320, processing unit 110 may determine at least oneindicator of vehicle position, for example, based on identified featuresfrom the acquired images (e.g., lane markers). At step 2330, processingunit 110 may use the at least one indicator to determine a relevance ofeach of the plurality of traffic lamp fixtures. At step 2340, processingunit 110 may determine a status of a traffic light included in arelevant traffic lamp fixture. For example, the status of the trafficlight may indicate that vehicle 200 should stop (e.g., due to a redlight) or can turn left (if there are no crossing pedestrians). Afterdetermining the status of a traffic light included in a relevant trafficlamp fixture, at step 2350, processing unit 110 may cause a systemresponse based on the determined status. These steps of process 2300 arediscussed in further detail below.

For example, at step 2310, image capture device 122 may acquire at leastone image of an area forward of vehicle 200. An “area forward of thevehicle” includes any geographical region located in front of a vehicle,relative to its moving direction. The region may include a junction (forexample intersections 2000 in FIGS. 20 and 2100 in FIG. 21), anintersection, a crossroad, a traffic circle, a street, a road, etc. Insome cases, the area may include a plurality of traffic lamp fixtureseach including at least one traffic light. A “traffic lamp fixture”includes any form of structure housing one or more light-producingdevices used to regulate traffic and/or to provide road-relatedinformation. In some cases, two or more traffic lamp fixtures may bejoined in a single traffic lamp assembly, but each traffic lamp fixturemay be associated with a different lane. A typical traffic lamp fixturemay include three circular traffic lights: a green traffic light, ayellow traffic light, and a red traffic light. A “traffic light”includes a device having at least one light source capable of displayinga distinctive color. In some cases, vehicle 200 may encounter anon-typical traffic lamp fixture. A non-typical traffic lamp fixture mayinclude one or more non-circular traffic lights having different colors.For example, a right-turn arrow traffic light, a left-turn arrow trafficlight, a public transportation traffic light, a pedestrian crossingtraffic light, a bicycle crossing traffic light, etc.

In some embodiments, processing unit 110 may receive the at least oneacquired image via data interface 128. As discussed in detail withreference to FIG. 5D, processing unit 110 may identify traffic lights inthe at least one acquired image. For example, processing unit 110 mayfilter objects identified in the at least one acquired image toconstruct a set of candidate objects, excluding those objects unlikelyto correspond to traffic lights. The filtering may be done based onvarious properties associated with traffic lights, such as shape,dimensions, texture, position (e.g., relative to vehicle 200), and thelike. In addition, processing unit 110 may analyze the geometry of thearea forward of vehicle 200. The analysis may be based on anycombination of: (i) the number of lanes detected on either side ofvehicle 200, (ii) markings (such as arrow marks) detected on the road,and (iii) descriptions of the area extracted from map data. For example,if lane markings defining a lane of travel are recognized on a road, anda traffic light fixture is within the boundaries of the lane markings,system 100 may conclude that the traffic light fixture is associatedwith the lane associated with the lane markings.

At step 2320, processing unit 110 may determine at least one indicatorof vehicle position from the images. An “indicator of vehicle position”includes any form of information related to the physical location of avehicle. The indicator of vehicle position may be derived from analyzingthe image data (e.g., the at least one acquired image). Additionally,the indicator of vehicle position may be derived from geographicposition data (e.g., GPS signals, local positioning signals, and/or mapdata) or from data indicative of the position of vehicle 200 relative toother vehicles on the road. In some embodiments, the indicator ofvehicle position may include the distance to at least one traffic lampfixture derived from the at least one image. In other embodiments, theat least one indicator of vehicle position may include a lane markerrecognized based on analysis of the at least one image. For example,processing unit 110 may conduct an analysis using information derivedfrom the at least one image to identify one or more lane markers. Usingthe identified lane markers, processing unit 110 may determine acorrespondence between the detected traffic lights and the lane vehicle200 is currently driving.

At step 2330, processing unit 110 (e.g., via relevancy determiningmodule 2202) may use the at least one indicator of the vehicle position,as determined at step 2320, to determine the relevancy of each of theplurality of traffic lamp fixtures to vehicle 200. In some embodiments,processing unit 110 may rank the relevancy of the traffic lamp fixturesidentified in the at least one acquired image. The traffic lamp fixturehaving the highest value of relevancy ranking may be determined to bethe relevant traffic lamp fixture. For example, in the situationillustrated in FIG. 20 the field-of-view of image capture device 122includes both traffic lamp fixture 2012 and traffic lamp fixture 2014,but by using one or more assessments, processing unit 110 may determinethat traffic lamp fixture 2014 has a higher relevancy ranking value. Oneway for processing unit 110 to determine that traffic lamp fixture 2014has a higher relevancy ranking than traffic lamp fixture 2012 is byassessing the distance of each of the plurality of traffic lamp fixtureswith respect to vehicle 200 a. For example, in the situation depicted inFIG. 20, traffic lamp fixture 2014 is closer than traffic lamp fixture2012. Thus, traffic lamp fixture 2014 is more likely to be relevant thantraffic lamp fixture 2012. Other ways to determine relevancy aredescribed above in reference to relevancy determining module 2202.

In some embodiments, the relevancy determination of each of theplurality of traffic lamp fixtures to vehicle 200 may include apreliminary examination to eliminate improbable traffic lamp fixtures.For example, when the at least one acquired image includes three closetraffic lamp fixtures and two distant traffic lamp fixtures, the twodistant traffic lamp fixtures may be classified as improbable to berelevant to vehicle 200. By eliminating improbable traffic lamp fixturesand ranking the relevancy of a subject of the traffic lamp fixturesidentified in the at least one acquired image, processing unit 110 maysave processing power. In some embodiments, the relevancy ranking maychange when vehicle 200 approaches the junction. For instance, theorientation of the traffic lamp fixture may change from different pointson the road, thus, the distance to the junction may impact theprobability that a given traffic lamp fixture is relevant to vehicle200. Accordingly, the relevancy ranking may be associated with aconfidence level, which may take into account factors, such as distanceto a junction, when assessing traffic lamp fixtures. Further, processingunit 110 may periodically or constantly update the relevancy ranking ofthe traffic lamp fixtures when the confidence level is below a certainpredetermined threshold.

Relevancy determining module 2202 may further use navigation informationpreviously stored within system 100, such as within memory 140/150, inorder to determine a traffic light relevant to vehicle 200. Based on adetermined 3D position of vehicle 200 and/or image capture devices122-126 as discussed above, relevancy determining module 2202 mayperform a registration between navigational map data and vehicle 200.Based on the determination of relative distance measurements asdescribed above, relevancy determining module 2202 may use thenavigational map data to determine the point in 3D space at whichvehicle 200 (via braking system 230) should brake in order to stop ateach detected traffic light fixture at the junction. According to the 3Dregistration result, relevancy determining module 2202 may determine alane assignment for each traffic light detected at the intersectionusing the navigational map data. Relevancy determining module 2202 maythen determine the lane assignments within system 100 and imageprocessor 190, then perform the registration.

At step 2340, processing unit 110 (e.g., via status determining module2204) may determine, based on the at least one acquired image, a statusof a traffic light included in at least one traffic lamp fixturedetermined to be relevant to vehicle 200. In some cases, the relevanttraffic lamp fixture may include a plurality of illuminated trafficlights, and the status of each traffic light may depend on the type ofthe traffic light. In one embodiment, the status of a traffic lightmeans simply the color it indicates, e.g., green, yellow, and red.System 100 may identify the status of a traffic light using a variety oftechniques. For example, system 100 may identify an area of one or moreimages that include a traffic light and perform an analysis of thepixels in the area to determine the colors of the pixels. Afteranalyzing at least a threshold number of pixels (e.g., two pixels, tenpixels, twenty pixels, etc.) in the area, system 100 may, for example,determine the color of the traffic light by finding an average value ofthe pixels in the area.

Additionally or alternatively, the status of the traffic light mayinclude the direction the traffic light refers to. In a typical trafficlight, a green color indicates that vehicle 200 is allowed to proceed.However, sometimes this information by itself is insufficient to decidewhether it is safe to drive in a certain direction, such as when a greenlight only authorizes a turn. One way to determine which direction thetraffic light refers to includes accessing a database that correlateseach traffic light with one or more directions. Another way includesidentifying, from the image data, the type of the traffic light anddetermining from the contextual situation the direction the trafficlight refers to. For example, in the second situation depicted in FIG.20 relative to vehicle 200 b, traffic lamp fixture 2024 may bedetermined as the relevant traffic fixture, but traffic lamp fixture2024 includes two illuminated traffic lights 2024 a and 2024 b.Accordingly, in step 2340, processing unit 110 may determine that thestatus of traffic light 2024 a is a red light for continuing straight,and the status of traffic light 2024 b is a green light for turningright. In some embodiments, the determination of the status of thetraffic light includes one or more of determining a location of thetraffic light within a relevant traffic lamp, determining whether thetraffic light is illuminated, determining a color of the traffic light,and determining whether the traffic light includes an arrow.

At step 2350, processing unit 110 (e.g., via system response module2206) may cause a system response based on the determined status. Insome embodiments, processing unit 110 may cause one or more systemresponses (e.g., two or more responses), including responses ofdifferent types. One type of system response may include navigationalresponses, the navigational response may include, for example, startingto drive, a change in acceleration, a change in velocity, applyingvehicle brakes, discontinuing cruise control, and the like. For example,these system responses may include providing control signals to one ormore of throttling system 220, braking system 230, and steering system240 to navigate vehicle 200. Another type of system response may includeinitiating a timer and/or incrementing a counter in order to providestatistical information about one or more driving sessions to the driverof vehicle 200. For example, the statistical information may indicatehow many times vehicle 200 has encountered red lights in a drivingsession, and/or the duration of a driving session that vehicle 200 spentwaiting at red lights. Another type of system response may includeproviding various notifications (e.g., warnings and/or alerts) to thedriver of vehicle 200. The warnings and/or alerts may include, forexample, announcing the color of a relevant traffic light and/or adistance to a junction. The notifications may be provided via speakers360 or via an associated display (e.g., touch screen 320).

Turn Lane Traffic Light Detection

System 100 may detect that a vehicle, such as vehicle 200, is travelingin a turn lane. A “lane” may refer to a designated or intended travelpath of a vehicle and may have marked (e.g., lines on a road) orunmarked boundaries (e.g., an edge of a road, a road barrier, guardrail, parked vehicles, etc.), or constraints. System 100 may operate tomake these detections and determinations based on visual informationacquired via one or more image capture devices, for example. In someembodiments, these detections and determinations may also be made atleast in part on map data and/or sensed vehicle position. In addition todetermining the status of vehicle 200 as being in a turn lane, system100 may recognize a traffic light associated with the lane, and may beconfigured to determine the status of the traffic light based onanalysis of road context information and determined characteristics ofthe traffic light.

FIG. 24 illustrates a vehicle 200 traveling on a roadway 800 in whichthe disclosed systems and methods for detecting an object in the roadwaymay be used. Vehicle 200 is depicted as being equipped with imagecapture devices 122 and 124; more or fewer cameras may be employed onany particular vehicle 200. As shown, roadway 800 may be subdivided intolanes, such as lanes 810 and 820. Lanes 810 and 820 are shown asexamples; a given roadway 800 may have additional lanes based on thesize and nature of the roadway. In the example of FIG. 24, vehicle 200is traveling in lane 820, and can be seen to be approaching intersection2402 with cross street 2404. Traffic in lane 820 at intersection 2402 isregulated by traffic light fixture 2406. System 100 may, as discussed indetail below, determine the status of traffic light fixture 2406 andcause a system response affecting operations of vehicle 200. Furtherillustrated on roadway 800 are warning lines 2410 leading to stop line2412.

Processing unit 110 may be configured to determine one or more laneconstraints associated with each of lanes 810 and 820, intersection2402, or cross street 2404 based on a plurality of images acquired byone or more of image capture device 122-126 that processing unit 110 mayreceive via data interface 128. According to some embodiments, the laneconstraints may be identified by visible lane boundaries, such as dashedor solid lines marked on a road surface. Additionally or alternatively,the lane constraints may include an edge of a road surface or a barrier.Additionally or alternatively, the lane constraints may include markers(e.g., Botts' dots). According to some embodiments, processing unit 110(via lane positioning module 2520, described in detail below) maydetermine constraints associated with lanes 810/820, intersection 2402,or cross street 2404 by identifying a midpoint of a road surface width,such as the entirety of roadway 800 or cross street 2404 or a midpointof one of lanes 810/820. Processing unit 110 may identify laneconstraints in alternative manners, such as by estimation orextrapolation based on known roadway parameters when, for example, linesdesignating road lanes such as lanes 810/820 are not painted orotherwise labeled. Processing unit 110 may also determine the physicalrelative distance between the various constraints and detected objects,such as traffic light lamps.

Distance estimation for a junction with traffic lights may be achallenge when developing autonomous vehicle or red traffic lightwarning systems, because the location of the traffic light may be afteror in the middle of the junction. For example, determining the 3Dposition for each traffic light at an intersection may imply a brakingpoint for the vehicle but the stop line may provide an accurateposition.

In the present system, detection of the constraints and pathways ofroadway 800, cross street 2404, or intersection 2402, as well asconstituent lanes 810/820 may include processing unit 110 determiningtheir 3D models via a camera coordinate system. For example, the 3Dmodels of lanes 810/820 may be described by a third-degree polynomial.In addition to 3D modeling of travel lanes, processing unit 110 mayperform multi-frame estimation of host motion parameters, such as thespeed, yaw and pitch rates, and acceleration of vehicle 200. Processingunit 110 may further determine a road elevation model to transform theinformation acquired from the plurality of images into 3D space. Onefeature of the present system is that a global model may be created forstatic objects at a traffic intersection simultaneously rather thancreating one standalone model for every detected object. System 100 maythus be configured to determine absolute distances for objects or laneconstraints at a distance, for example, within about one hundred meters.In some cases, system 100 may be configured to determine absolutedistances for objects or lane constraints at other distances (e.g.,within 125 meters, within 150 meters, etc.). Further, system 100 may beconfigured to determine which traffic light fixture (e.g., traffic lightfixture 2406) is relevant to vehicle 200 within a distance of abouteighty meters from stop line 2412.

Accurate distance estimation to the traffic light(s) at an intersectionmay be achieved through processes referred to as expansion and scaling.The expansion process may use relative distances to determine a distancefrom vehicle 200 to an object, such as stop line 2412. For example, iftwo traffic lamp fixtures are detected in the image data, and thedistance between them increases by 5% when the vehicle moves a distanceof ten meters, then the system calculates that the distance is 200meters.

FIG. 25 is an exemplary block diagram of memory 140 and/or 150, whichmay store instructions for performing one or more operations consistentwith disclosed embodiments. As illustrated in FIG. 25, memory 140/150may store one or more modules for performing the object detection andresponses described herein. For example, memory 140/150 may store atraffic light detection module 2510, a lane positioning module 2520, andan action module 2530. Application processor 180 and/or image processor190 may execute the instructions stored in any of modules 2510-2530included in memory 140/150. One of skill in the art will understand thatreferences in the following discussions to processing unit 110 may referto application processor 180 and image processor 190 individually orcollectively. Accordingly, steps of any of the following processes maybe performed by one or more processing devices.

Traffic light detection module 2510 may store instructions which, whenexecuted by processing unit 110, may detect the presence and status of atraffic light fixture, such as traffic light fixture 2406. As will bediscussed below in association with FIGS. 26A, 26B, and 27, trafficlight detection module 2510, along with image processor 190, may performimage processing on one or more images acquired by one or more of imagecapture devices 122-126. Traffic light detection module 2510 may furtherdetermine the status of traffic light fixture 2406, including adetermination of whether one of the lights within traffic light fixture2406 includes an arrow. Traffic light detection module 2510 maydetermine other information relevant to the status of traffic lightfixture 2406, including but not limited to whether any of the trafficlights associated within traffic light fixture 2406 are illuminated (ineither a solid or blinking manner), determining positions of the trafficlights within the traffic lamp fixture (i.e. a horizontal orientation oftraffic lights versus a vertical orientation), or determining a colorassociated with the traffic light. In some embodiments, traffic lightdetection module 2510 may store information determined for a particularheading, a particular intersection 2402, and/or a particular trafficlight fixture 2406 within processing unit 110, such as in memory140/150. In these embodiments, previously determined and savedinformation may be used in the future when vehicle 200 returns to thesame intersection.

In some embodiments, a blinking traffic light may be determined throughimage analysis of multiple images acquired at a predetermined or knowncapture rate (e.g., analyzing images acquired 1 second, 1.5 seconds, 2seconds, etc., apart). For example, system 100 may analyze image data toidentify a pattern in illumination among a plurality of images. System100 may further determine a region of a captured image determined to bewithin the boundaries of a particular traffic light of interest in atraffic light fixture. System 100 may then determine the color of atraffic light through pixel analysis in the region determined to bewithin the boundaries of the traffic light of interest.

Lane positioning module 2520 may store instructions which, when executedby processing unit 110, may assist system 100 in determining a positionof vehicle 200. In some embodiments, determining the position of vehicle200 may include determining at least one indicator of vehicle position,either via a visual determination or through analysis of the at leastone image received from image capture devices 122-126 via data interface128. In these embodiments, the at least one indicator of vehicleposition may include a distance from the vehicle to one or more laneconstraints or lane markers associated with the current lane in whichthe vehicle is traveling, or to markers associated with an intersection,such as warning lines 2410 and stop line 2412. The distance from thevehicle to a location or object may be determined based on, for example,one or more of an analysis of image data, GPS information, or data froma position sensor. Further, the at least one indicator of vehicleposition may include an arrow associated with the traffic lamp fixturehaving at least one associated traffic light. Additionally oralternatively, the at least one indicator of vehicle position mayinclude a vehicle location acquired by GPS or a like coordinate system.In some embodiments, lane positioning module 2520 may store laneconstraint information determined for a particular roadway 800 and itslanes 810/820. For example, processing unit 110 may store theinformation in memory 140/150. In these embodiments, previouslydetermined and saved information may be used in the future when vehicle200 returns to the same intersection. For example, GPS information maybe used to determine that vehicle 200 has returned to the sameintersection.

Consistent with disclosed embodiments, lane positioning module 2520 mayuse the information from the at least one indicator of vehicle positionto determine if a system response changing the operation of vehicle 200is required or recommended. Additionally or alternatively, lanepositioning module 2520 may receive information from other modules(including position sensor 130) or other systems indicative of thepresence of other features in the image data, such as additionalvehicles, curvature of the road, etc.

Action module 2530 may store instructions which, when executed byprocessing unit 110, may assist system 100 in taking one or more actionsrelative to the operation of vehicle 200 based on information receivedfrom one or more sources, such as position sensor 130, image processor190, traffic light detection module 2510, or lane positioning module2520. In some embodiments, action module 2530 may receive information(from, e.g., traffic light detection module 2510) regarding the statusof a traffic light at an intersection, such as traffic light fixture2406 at intersection 2402 discussed above. Other information received byaction module 2530 may include a determination of whether the vehicle isin a turn lane, and whether the traffic light fixture 2406 includes anarrow.

Based on this received information, action module 2530 may then cause asystem response affecting the operational status of vehicle 200, such ascausing system 100 to provide control signals to one or more ofthrottling system 220, braking system 230, and steering system 240 tonavigate vehicle 200 (e.g., by accelerating, turning, etc.). Forexample, in some embodiments, system 100 may determine that the turnlane traffic light authorizes vehicle 200 to make a turn. In theseembodiments, action module 2530 may send an instruction to steeringsystem 240, and steering system 240 may execute the instruction to turnvehicle 200 through intersection 2402 into a new lane of travelassociated with cross street 2404. In other embodiments, the systemresponse initiated and executed by action module 2530 may include any orall of providing a visual or audible notice to the operator of thevehicle, applying vehicle brakes, discontinuing a cruise controlfunction, or initiating one or more automated turning maneuvers.

Additionally or alternatively, action module 2530 may send instructionsto other systems associated with vehicle 200, such as braking system230, turn signals, throttling system 220, etc. In some embodiments,action module 2530 may instead provide a human operator of the vehiclewith audio, visual, or tactile feedback representative of theinformation gathered from the relevant systems and/or sensors. The humanoperator may then act on this feedback to turn the vehicle.

FIG. 26A provides an annotated view of the situation depicted in FIG.24. Vehicle 200 is once again traveling in lane 820 of roadway 800, andis approaching intersection 2402 with cross street 2404. Vehicle 200 isagain equipped with image capture devices 122 and 124; more or fewerdevices may be associated with any particular vehicle 200. Forsimplicity of illustration, roadway 800 in FIG. 26A is depicted as aone-way street oriented from the top to the bottom of the page with twotravel lanes 810 and 820. Cross street 2404 is a two-way street orientedleft-right across the page with one travel lane going in each direction.

To the left side of roadway 800 at intersection 2402 is traffic lightfixture 2406. Painted on the surface of lane 820 in front of vehicle 200is lane arrow 2408, indicating that lane 820 is a left turn lane. Alsopainted or otherwise affixed on the surface of roadway 800 are warninglines 2410 leading to stop line 2412.

Consistent with disclosed embodiments, system 100 may be configured todetermine whether vehicle 200's travel lane approaching an intersection(here, lane 820) is a turn lane; determine whether a traffic lightfixture (here, traffic light fixture 2406) regulates the intersection;determine the status of a traffic light in the traffic light fixture;and determine whether that traffic light includes an arrow. This processwill be described in further detail below in association with FIG. 27and process 2700.

Briefly, system 100 associated with vehicle 200 may determine theposition of vehicle 200 within roadway 800 via one or more of processingunit 110, position sensor 130, traffic light detection module 2510, orlane position module 2520. Additionally or alternatively, system 100 maygather information from at least one indicator of vehicle position. Asdiscussed above in association with lane positioning module 2520, the atleast one indicator of vehicle position may include a distance fromvehicle 200 to one or more lane constraints or lane markers associatedwith the current lane in which the vehicle is traveling (such as lane820), or to markers associated with an intersection 2401, such aswarning lines 2410 and stop line 2412. Further, the at least oneindicator of vehicle position may include an arrow associated withtraffic light fixture 2406. Additionally or alternatively, the at leastone indicator of vehicle position may include a vehicle locationacquired by GPS or a like coordinate system.

In the illustration of FIG. 26A, multiple indicators of the position ofvehicle 200 are present. One or more of image capture devices 122-126associated with vehicle 200 may be configured to capture a plurality ofimages of the area in front of vehicle 200 that may assist indetermining a position of vehicle 200. For example, the images mayinclude lane arrow 2408, which indicates that lane 820 is a left turnlane for intersection 2402. The images may also include traffic lightfixture 2406, and may indicate that one or more of the individual lampswithin traffic light fixture 2406 includes an arrow suggestive of a turnlane situation. One or more of image capture devices 122-126 may furthercapture images of warning lines 2410 or stop lines 2412 associated witheither roadway 800 or cross street 2404. Further, system 100 via lanepositioning module 2520 may determine a position of vehicle 200 withinthe turn lane by measuring distance s₁ from a surface of vehicle 200 tostop line 2412. Still further, system 100 may implement an opticalcharacter recognition (OCR) process to obtain text included in one ormore captured images (e.g., text from signs and/or road markings).System 100 may then use the text information as part of or as the basisof determining whether vehicle 200 is within a turn lane. For example,system 100 may identify certain words indicative of a turn lane (e.g.,“turn, “right,” left,” etc.).

The measurement(s) may be direct measurements, such as via positionsensor 130, or may be determined via analysis of captured image data byimage processor 190 and may be used in connection with map data. Themeasurement(s) may be measured from any portion of the interior orexterior vehicle 200, including but not limited to the front of vehicle200, a portion of vehicle 200 such as a headlight or front licenseplate, a position as-installed of image capture devices 122-126, adetermined centroid of vehicle 200, the rear of vehicle 200, one or morewindshields or mirrors associated with vehicle 200, wheels of vehicle200, right or left sides or windows of vehicle 200, a point associatedwith the roof of vehicle 200, or a point associated with the chassis ofvehicle 200.

In some embodiments, determining the distance from vehicle 200 totraffic light fixture 2406 may be sufficient to assist processing unit110 in calculating a braking distance or other such measurement. Inother embodiments, however, traffic light fixture 2406 may be locatedpast the point at which vehicle 200 would be required to stop. In theseembodiments, the distance to the intersection may be determined by usingone or more of warning lines 2410 or stop line 2412.

As discussed above in association with FIG. 24, these distances may becalculated using an estimation process employing expansion and scaling.At an intersection 2402 with two traffic light fixtures, for example,system 100 may measure the relative distance between the two fixturesover a period of time as captured in the image data, and then use thatrelative distance to estimate the distance to stop line 2412. In someembodiments, these measurements may be repeated over time for increasedprecision. Precision may also be increased by other methods; forexample, if three or four traffic light fixtures are situated atintersection 2402 relative distances may be calculated between each ofthe fixtures and averaged. Additionally or alternatively, the distancemay be estimated using a Kalman filter, as discussed above.

System 100 may still need additional information or inputs to determinethe location of a stop line 2412 relative to an intersection 2402 and/ortraffic light fixtures 2406. In some embodiments, system 100 may usepreviously-stored map data gathered from prior trips to theintersection. This information may be received from traffic lightdetection module 2510, or lane positioning module 2520. A distancemeasure Z may also be derived from the image data using the equationZ=fW/w

where W is the known distance between two traffic light fixtures 2406, wis the distance in the image data in pixels, and f is the focal lengthof the particular image capture device 122-126 in pixels.

In FIG. 26A, inset 2602 shows the two statuses of traffic light fixture2406 relevant to the example. Based on image data from at least oneimage received via data interface 128 from image capture devices122-126, system 100 may be configured to determine the status of trafficlight fixture 2406. This determination will be described in furtherdetail below in association with FIG. 27 and process 2700. Uponapproaching intersection 2402, system 100 may first determine thattraffic light fixture 2406 is displaying a solid red light, as seen inimage 1 of inset 2602. In the example of FIG. 26A, lane 820 is a leftturn lane, so unless roadway 800 is located in a jurisdiction where leftturns on red are legal (or if cross street 2404 were configured as a oneway street with both lanes flowing from right to left on the page),vehicle 200 must stop at stop line 2412 and remain stationary untiltraffic light fixture 2406 is determined to have changed. In alternativeembodiments (not shown), vehicle 200 may be traveling in a lane that isdetermined to be a right turn lane, and assuming that right terms on redare legally permissible in that jurisdiction, system 100 may cause asystem response enabling vehicle 200 to turn right while traffic lightfixture 2406 remains red.

Subsequently, as seen in image 2 of inset 2602, traffic light fixture2406 shifts from a solid red light to a solid green arrow indicating aprotected left turn. System 100 may be configured to detect not onlythat the color of the light has changed, but also that an arrow hasappeared within the lamp based on pixel-level image analysis by imageprocessor 190 of acquired images of traffic light fixture 2406. In someembodiments, in order for system 100 to detect that the color of atraffic light has changed, system 100 may analyze captured images over aperiod of time (e.g., 1 second, 5 seconds, 10 seconds, 15, seconds, 20seconds, 30 seconds, 1 minute, etc.) in order to recognize the statuschange of the traffic light. System 100 may further determine oncetraffic light fixture 2406 changes to display the green arrow whether itis safe or authorized to proceed in the direction indicated by thestatus of the light. Even though the light may show a green arrow,various situations (not shown) may require vehicle 200 to further delaybefore completing the turn. For example, a pedestrian may be crossingcross street 2404. Although roadway 800 in FIG. 26A is a one-way street,in alternative embodiments (such as that described below and shown inFIG. 26B), the road may be a two-way street and vehicle 200 may furtherassess whether a second vehicle coming from the opposite direction ismaking a right turn on red onto cross street 2404 from the otherdirection. Further, other obstacles may prevent a timely left turn; forexample, a set of railroad tracks may be oriented parallel to roadway800 to its left side, and although a left turn green arrow may beilluminated, barriers associated with the train track may be down toallow a train to pass through. These examples are not intended to belimiting. Once system 100 determines that both traffic light fixture2406 and the situation at the intersection enables a safe turn, system100 via action module 2530 may cause a system response involving one ormore of throttling system 220, braking system 230, steering system 240,turn signals, and/or other subsystems and modules that automaticallycauses vehicle 200 to turn left.

FIG. 26B illustrates an additional intersection scenario consistent withdisclosed embodiments. In the example of FIG. 26B, vehicle 200 (hererepresented as vehicle 200 e) is now traveling in lane 810 of roadway800, and is approaching intersection 2402 with cross street 2404.Vehicle 200 is again equipped with image capture devices 122 and 124;more or fewer devices may be associated with any particular vehicle 200.Unlike the illustration of FIG. 26A, roadway 800 is now a two-way streetin FIG. 26B, and traffic may proceed in both directions. Cross street2404 remains a two-way street oriented left-right across the page withone travel lane going in each direction.

To the right side of roadway 800 at intersection 2402 is traffic lightfixture 2604. Painted on the surface of lane 810 in front of vehicle 200is lane arrow 2408, indicating that lane 810 is a left turn lane. Asbefore, consistent with disclosed embodiments, system 100 may beconfigured to determine whether vehicle 200's travel lane approaching anintersection (this time, lane 810) is a turn lane; determine whether atraffic light fixture (here, traffic light fixture 2604) regulates theintersection; determine the status of the traffic light; and determinewhether that traffic light includes an arrow.

System 100 may operate to make these determinations in much the samemanner as described above. However, in the example of FIG. 26B, inset2606 demonstrates that as vehicle 200 e approaches intersection 2402,traffic light fixture 2604 is displaying a flashing or blinking yellowlight. The non-solid nature is indicated in inset 2606 by the dashedlines of the arrow. As may be familiar to one of ordinary skill in therelevant art, some jurisdictions use blinking yellow lights toessentially serve the same functions as “Yield” or “Stop” signs duringcertain times of the day, while still permitting the lanes of travel tobe fully stopped with a red light or fully authorized to proceed with agreen light.

As outlined above, system 100 may determine that traffic light fixture2604 is displaying a blinking yellow light, and may further determinethat lane 810 is a turn lane. With the situation depicted in FIG. 26B,however, system 100 may be required to make additional determinations inorder for vehicle 200 e to successfully traverse intersection 2402. Asone of ordinary skill would understand, a blinking yellow lightauthorizes an approaching vehicle to turn at will, but only if it issafe to do so. Drivers are expected to use caution at the intersectionand determine that no potential collisions or other hazards would resultfrom making the turn too early.

One such determination that system 100 may make is whether one or morevehicles are approaching the intersection from the other direction. Forexample, while travel lanes traveling one direction of a roadway mayhave a traffic light indicating a blinking yellow light in a turn lane,traffic traveling the opposite direction may have a green light toproceed straight through the intersection. Accordingly, a collisioncould occur if the operator of the vehicle with the blinking yellowsignal fails to yield. In the example of FIG. 26B, a second vehicle 200f is indeed passing straight through intersection 2402. System 100 maybe configured to detect the presence of vehicle 200 f via image datacaptured from one or more of image capture devices 122-126, as well asdetermine that vehicle's location within the roadway/intersection andits velocity using one or more of position sensor 130, image processor190, traffic light detection module 2510, or lane positioning module2520. Upon detecting and analyzing vehicle 200 f, system 100 (via actionmodule 2530) may cause braking system 230 to remain engaged untilvehicle 200 f has passed.

Once no additional vehicles are approaching the intersection from theopposite direction, system 100, via action module 2530, may cause asystem response based on that determination. The system response mayinclude one or more of providing feedback to an operator of vehicle 200,applying vehicle brakes via braking system 230, discontinuing cruisecontrol, or initiating an automated turning maneuver onto cross street2404 using one or more of throttling system 220, braking system 230, orsteering system 240. First, however, system 100 may analyze image datafor the presence of any additional obstacles or objects associated withintersection 2402 or cross street 2404. For example, in the illustrationof FIG. 26B, pedestrian 2608 can be seen walking across cross street2404, since the other lane of roadway 800 has a green light in thatdirection. System 100 may detect the presence of pedestrian 2608 viaimage processor 190 by analyzing one or more of the plurality of imagescaptured by one or more of image capture devices 122-126. As describedabove, other subsystems and modules of system 100 may coordinate todetect the velocity of pedestrian 2608, and may notify system 100 whenpedestrian 2608 is no longer in the roadway and it is safe to turn.

FIG. 27 illustrates a process 2700 for detecting the status of a trafficlight, consistent with disclosed embodiments. Steps of process 2700 maybe performed by one or more of processing unit 110, image acquisitionunit 120, position sensor 130, image processor 190, traffic lightdetection module 2510, lane positioning module 2520, or action module2530.

At step 2710, process 2700 may include acquiring, using at least oneimage capture device 122, 124, and/or 126, a plurality of images of anarea forward of vehicle 200. The area may contain one or more trafficlamp fixtures having at least one traffic light, such as traffic lightfixtures 2406 and 2604 described above. Processing unit 110 may receivethe plurality of images from the image capture device(s) through datainterface 128. The plurality of images may then be processed in realtime by image processor 190 of processing unit 110. Image processor 190may process at least one image captured by the at least one imagecapture device(s), as well as additional images captured at later pointsin time.

At step 2720, process 2700 may determine, based on at least oneindicator of vehicle position, whether vehicle 200 is traveling in aturn lane. As described generally above, various indicators of positionmay be used to determine whether a travel lane is a turn lane. In someembodiments, the position indication may include a vehicle locationacquired by GPS. In other embodiments, system 100 may recognize lanemarkers or constraints, such as lane arrow 2408 indicating that the laneis a turn lane. Alternatively, image processor 190 may determine that anarrow is associated with the traffic lamp fixture regulating the lanetravel. Additionally, measurements of distance may be made, usingvarious elements of system 100. For example, position sensor 130 and/orlane positioning module 2520 may send information to processing unit 110regarding the relative position of vehicle 200 within lane 810/820,including how far vehicle 200 is from intersection 2402 (distance s₁described above). The end of the lane may be based on a measureddistance or a visual determination (e.g., by analyzing one or more imageimages acquired by one or more of image capture devices 122-126 anddetecting the presence of warning lines 2410 or stop line 2412).

At step 2730, process 2700 may perform image processing on the at leastone image received via data interface 128 to determine whether an arrowexists in the image data associated with the traffic light. Trafficlight detection module 2510 and/or image processor 190 may be configuredto analyze the pixels of the image corresponding to the locationsrepresentative of the individual traffic lights within the traffic lampfixture. One or more image processing algorithms may be applied to thepixels to determine if either a solid or blinking/flashing arrow ispresent in the relevant regions. Further examples of how system 100 mayanalyze one or more images to determine whether an arrow is displayed bya traffic light are discussed below in connection with FIGS. 28-30B.

At step 2740, process 2700 may determine a status of the traffic light.As discussed above, determining the status of the traffic light mayinclude, but not be limited to, determining whether the traffic light isilluminated (and/or functional at all), determining a position of theilluminated traffic light within the traffic lamp fixture, ordetermining a color associated with the traffic light (i.e. red, yellow,or green). This list is intended as exemplary only, and any informationrelevant to the traffic light may be determined and processed by system100 and processing unit 110. For example, the system may be configuredto determine whether the traffic light is solid or flashing based onimage processing over a series of acquired images. Image processor 190may determine in these embodiments that the same traffic light withinthe fixture is illuminated in one image, darkened in the next in thesequence, and then immediately illuminated again; this would indicatethat the particular light is flashing. In some embodiments, system 100may be configured to recognize traffic light situations particular tothe specific jurisdiction in which the vehicle is traveling. Thisinformation may be pre-programmed into system 100, such as in a databasestored within memory 140/150 (not shown).

At step 2750, process 2700 may cause a system response via system 100impacting the operation of vehicle 200. The response initiated by system100 may be based on any or all of the determination of the status of thetraffic light, whether the traffic light includes an arrow, and whetherthe vehicle is in a turn lane. As discussed above, the particularaction(s) initiated by system 100 via action module 2530 may include,but not be limited to, providing a visual or audible feedback notice toan operator of vehicle 200 regarding the traffic light, applying vehiclebrakes via braking system 230, discontinuing a cruise control function,or proceeding to initiate an automated turning maneuver and proceedingthrough the intersection using one or more of throttling system 220,braking system 230, or steering system 240.

Additionally, in some embodiments as part of the function of system 100,action module 2530 and/or processing unit 110 may be further configuredto determine whether the turn lane in which the vehicle is located isauthorized to proceed in the direction indicated by the determinedstatus of the traffic light. In other words, even if the traffic lightindicates that a particular system response is legally permissible,system 100 may provide the occupants of vehicle 200 an extra level ofsafety protection by further determining whether such an action isactually possible and safe. For example, as discussed above inassociation with FIG. 26B, system 100 and image processor 190 may beconfigured to detect the presence of other obstacles associated with theroadway(s) or the intersection, such as pedestrians present in thestreet or the presence of other vehicles. In some embodiments, the othervehicles may be proceeding normally through the intersection, such as inthe scenario depicted in FIG. 26B where vehicle 200 e has a blinkingyellow light. In these embodiments, vehicle 200 e could theoreticallyturn onto cross street 2404, but system 100 may further account for thefact that vehicle 200 f is proceeding on a green light through theintersection. Alternatively, a second vehicle may be present in anunusual or illegal manner, such as if the operator of the other vehicleran through a stop sign or a red light, or if a vehicle has becomedisabled in the intersection or one of the roadways.

In still other embodiments, determining whether the turn lane in whichthe vehicle is located is authorized to proceed in the directionindicated by the determined status of the traffic light may furthercomprise determining whether to actually follow the indicated status ofthe light based on a predetermined destination for the vehicle. Inembodiments involving autonomous vehicles, it may be presumed that theoperators and/or passengers have indicated their desired destination tothe processing unit 110 of system 100. Accordingly, using GPS locationtechnology or other such information, system 100 may be configured todetermine if making the turn indicated by a traffic light is actuallyrelevant to the predetermined destination. For example, in someembodiments the vehicle 200 may have ended up in the turn laneunintentionally, and proceeding to the predetermined destination mayinvolve merging to an adjacent lane (with the help of lane positioningmodule 2520) and continuing to proceed straight through theintersection. Accordingly, system 100 may be configured to determinewhether to turn left, turn right, or continue straight through theintersection based on this predetermined destination information.

Traffic Light Detail Detection

In some embodiments, system 100 may provide additional functionalityrelated to the recognition of traffic signals. For example, system 100may provide a traffic light detail detection function to providefeedback based on analysis of the physical features of a traffic light.To implement such functionality, processing unit 110 may process imagescaptured by at least one of image capture devices 122, 124, 126 andemploy a super resolution technique to recognize a specific feature of atraffic light in the images, where the specific feature may indicateuseful information to system 100. For example, system 100 may recognizeand distinguish between round and arrow shaped signals of a trafficlight that have been illuminated.

In some embodiments, memory 140 and/or 150 may store instructionsprogrammed such that upon execution by a processing device will providethe traffic light detail detection function. As shown in FIG. 28, memory140 and/or 150 may store an image alignment module 2810, a pixelexpansion module 2820, an image analysis module 2830, and a database2840. Image alignment module 2810 may store instructions for aligning aplurality of images captured by image capture devices 122, 124, 126.Pixel expansion module 2820 may store instructions for increasing theresolution of at least a portion of an image to allow images to bescaled to a standard size. Image analysis module 2830 may storeinstructions for analyzing the images to detect and identify features ofthe image. Database 2830 may be configured to store data associated withproviding traffic light detail detection functionality and provide datawhen requested. Further, image alignment module 2810, pixel expansionmodule 2820, and image analysis module 2830 may store instructionsexecutable by one or more processors (e.g., processing unit 110), aloneor in various combinations with each other. For example, image alignmentmodule 2810, pixel expansion module 2820, and image analysis module 2830may be configured to interact with each other and/or other modules ofsystem 100 to perform functions consistent with disclosed embodiments.

Database 2840 may include one or more memory devices that storeinformation and are accessed and/or managed through a computing device,such as processing unit 110. In some embodiments, database 2840 may belocated in memory 150, as shown in FIG. 28. In other embodiments,database 2840 may be located remotely from memory 150, and be accessibleto other components of system 100 (e.g., processing unit 120) via one ormore wireless connections (e.g., a wireless network). While one database2840 is shown, it should be understood that several separate and/orinterconnected databases may make up database 2840. Database 630 mayinclude computing components (e.g., database management system, databaseserver, etc.) configured to receive and process requests for data storedin memory devices associated with database 2840 and to provide data fromdatabase 2840 (e.g., to processing unit 110).

In some embodiments, database 2840 may be configured to store dataassociated with providing traffic light detail detection functionality.For example, database 2840 may store data, such as images, maps,algorithms, sets of values, or the like, which may allow processing unit110 to identify information detected in an image. For example, database2840 may store a set of average pixel values corresponding to each of around signal and an arrow signal of a traffic light.

In some embodiment, system 100 may provide feedback based on a detail ofa traffic light detected in one or more capture images. For example,system 100 may be configured to differentiate between colors, shapes,and positions of a signal associated with a traffic light in order toprovide specific feedback based on a meaning of the detected detail(e.g., an arrow signal controls turning movement, while a round signalcontrol straight forward movement).

FIGS. 29A and 29B depict exemplary objects that may be detected bysystem 100. FIG. 29A depicts a traffic lamp fixture 2900A that includesa round signal 2910. FIG. 29B depicts a traffic lamp fixture thatincludes an arrow signal 2920. While both objects include trafficlights, they include different details that indicate different messages.In an exemplary embodiment, system 100 may be configured to analyze thepixels of one or more images to identify differentiating features, suchas a round signal 2910 and arrow signal 2920.

FIG. 29C depicts an exemplary portion of an image 2915 of round signal2910, including a plurality of pixels 2930. System 100 may be configuredto analyze pixels 2930 to identify an object and/or detail found in theimage. For example, image analysis module 2830 may be configured toidentify features such as brightness, color, contrast, tint, opacity,and the like, associated with each pixel 2930, and determine an objectand/or detail associated with the image based on the pixel analysis.Each feature that image analysis module 2830 identifies may be given avalue. For example, image analysis module 2830 may assign a value of oneto pixels that are identified as the color red and a value of zero toany other pixel.

In some embodiments, system 100 may be configured to use a superresolution technique to allow for a more accurate analysis of an objectin plurality of images. For example, pixel expansion module 2820 maymodify image 2915 to divide one or more of pixels 2930 into a subset ofpixels 2940. While only some pixels 2930 are shown as expanded intopixels 2940 in FIG. 29C, it should be understood that any portion or allof pixels 2930 may be divided in this way. Thus, pixel expansion module2820 may increase the number of pixels associated with a particularportion of an image.

In an exemplary embodiment, pixel expansion module 2820 may increase thenumber of pixels associated with a particular object in an image suchthat the section of the image including the object includes a selectednumber of pixels (e.g., a standard number of pixels). In this way, evenif a plurality of images include the same object at different sizes(e.g., because the images were taken at different distances from theobject), the images can be modified such that the portions of the imagesthat include the object include the same number of pixels, which can bedirectly compared and averaged to more reliably determine a feature(e.g., color) of the part of the object located within the pixel.

Scaling selected portions of images to associate the same number ofpixels with an object in each image may allow the images to be directlycompared (e.g., compare the value of one pixel in an image with the samepixel in another image). In this way, multiple images may be consideredand analyzed by image analysis module 2830, thus increasing accuracy ascompared to analysis of one image. For example, the super resolutiontechnique may increase the likelihood that image analysis module 2830may correctly distinguish between round signal 2920 and arrow signal2930 in images.

FIG. 29D depicts an exemplary portion of an image 2925 of arrow signal2920. As described above with respect to image 2915, image 2925 may bedivided into pixels 2930, which each may be expanded into a subset ofpixels 2940. As can be seen in FIGS. 29C and 29D, pixels 2930 differaround the edges of the respective signals 2920, 2930. Image analysismodule 2830 may be configured to identify one or more characteristics ofpixels 2930 to determine whether the pixels correspond to round signal2910 or arrow signal 2920. Expansion of pixels 2930 into pixels 2940 mayallow multiple images to be aligned and considered, thus increasing thereliability of detection.

FIG. 30A is a flowchart showing an exemplary process 3000A fordetermining whether a traffic light includes an arrow, consistent withdisclosed embodiments. In an exemplary embodiment, system 100 mayperform process 3000A to determine whether a portion of a traffic lightdetected in a plurality of images includes an arrow signal. An arrowsignal (or “arrow”) may be a type of signal of a traffic light in whicha visual component of the signal takes the shape of an arrow. The arrowmay be a signal indicating that a vehicle turn (e.g., a turn in adirection of the arrow) is allowable at that time.

At step 3002, processing unit 110 may receive a plurality of images of atraffic light. For example, one or more of image capture devices 122,124, 126 may acquire a plurality of images of an area forward of vehicle200. In some instances, such as when vehicle 200 is approaching ajunction, the area forward of vehicle 200 may include one or moretraffic lamp fixtures that include at least one traffic light. In theseinstances, one or more of image capture devices 122, 124, 126 mayacquire a plurality of images of the traffic light. Processing unit 110may receive the plurality of images from image capture devices 122, 124,126 through data interface 128.

Processing unit 110 may analyze the received plurality of images. Insome embodiments, processing unit 110 may analyze the plurality ofimages to determine that the plurality of images include a trafficlight. For example, processing unit 110 may recognize one or moreobjects as a candidate for being a traffic light and perform one or moreprocesses (e.g., image matching) to determine whether the candidatecorresponds to a traffic light. It should be understood, however, thatprocessing unit 110 may perform process 3000A without the distinct stepof identifying an object as a traffic light. In addition, processingunit 110 may analyze portions of the plurality of images to determinecharacteristics of the traffic light. These characteristics may helpsystem 100 determine a response to the traffic light (e.g., the light isgreen—move through the junction, the light is red—stop at the junction).

At step 3004, processing unit 110 (e.g., via image alignment module2810) may align areas of the plurality of images that correspond to thetraffic light. For example, processing unit 110 may use an area of atraffic light as a common point to align the plurality of images. Thealignment may allow processing unit 110 to analyze a common area ofmultiple images.

In order to align the areas of the plurality of images that correspondto the traffic light, processing unit 110 may select a trait orcharacteristic of a traffic light that may allow for an area of interestto be identified. In an exemplary embodiment, processing unit 110 mayalign areas of the plurality of images corresponding to the trafficlight based on a determined center of brightness (e.g., of a lit signalof the traffic light). In some embodiments, the center of brightness maybe determined by finding, for example, a center of mass of brightness ofa traffic light. In other embodiments, processing unit 110 may usecolor, shape, position, or any other trait or characteristic of atraffic light to align the plurality of images. Further, in someinstances, processing unit 110 may rotate the areas of the plurality ofimages to align the images in the plane of the image (e.g., to match therotational position of the areas).

As described above, the aligned areas of the plurality of images maycorrespond to an area of interest of the traffic light. For example, byaligning the plurality of images at areas corresponding to the trafficlight based on brightness, a lit signal of the traffic light may beidentified and placed within the aligned areas. A lit signal mayindicate to a viewer an action that is acceptable at that time (e.g.,stop, go, turn, etc.). Therefore, aligning areas that correspond to alit signal may help processing unit 110 isolate an area of interest ofthe traffic light.

In step 3006, processing unit 110 (e.g., via pixel expansion module2820) may expand pixels of the plurality of images within the alignedareas to obtain a standard size (e.g., the same number of pixels). Forexample, processing unit 110 may perform an image modification processin which processing unit 110 replaces each pixel corresponding to thealigned areas of the plurality images with a plurality of pixels. Inthis way, the aligned areas of one or more images may be expanded to astandard size. The expansion of the pixels may allow for directcomparison of pixels between the plurality of images.

In some embodiments, processing unit 110 may expand the pixels of theplurality of images within the aligned areas by replacing each pixelwith a matrix of pixels. Each pixel in the matrix of pixels may beattributed with the same value as the parent pixel. In this way, theexpanded pixels may also include a value associated with a feature ofthe image at the pixel (e.g., color, brightness, etc.).

In one embodiment, processing unit 110 may expand each pixel within thealigned areas into a 3×3 matrix of pixels. In another embodiment,processing unit 110 may expand each pixel within the aligned areas intoa 4×4 matrix of pixels. In other embodiments, other matrix sizes may beused (e.g., 5×5, 6×6, 8×8, etc.).

In some embodiments, pixel expansion module 2820 may expand pixels basedon a scaling factor (e.g., 2, 2.5, 3, 4, 5, etc.) that transforms eachpixel into a number of pixels corresponding to the scaling factor. Pixelexpansion module 2820 may determine a scaling factor based on a size ofan object in an image and a standard size that is used. In this way, theobjects in the images may be standardized to include the same number ofpixels.

Processing unit 110 may analyze the pixels across the plurality ofimages. For example, in step 3008, processing unit 110 (e.g., via imageanalysis module 2830) may determine a set of average pixel valuesincluding an average pixel value for each pixel, including expandedpixels. As described above, expanded pixels include values associatedwith a parent pixel. The pixel values may be any value that defines aselected pixel and allows processing unit 110 to determine acharacteristic of the traffic light in the aligned areas. For example,the pixel values may correspond to a color or brightness valueassociated with each pixel. In another embodiment, the pixel values maybe binary values indicating whether each pixel (or pixel matrix average)includes a color or brightness value above a threshold value. Processingunit 110 may determine an average pixel value for each pixel across allof the plurality of images.

Processing unit 110 may use the pixel values to determine acharacteristic of the traffic light in the aligned areas. For example,in step 3010, processing unit 110 may determine whether the trafficlight includes an arrow based on the set of average pixel values. In oneexample, processing unit 110 may compare the set of average pixel valuesto stored criteria (e.g., information stored in database 2840) todetermine whether the pixel values correspond to an arrow. In anotherexample, processing unit may map the average pixel values and compare itto a stored map to determine whether the pixel values correspond to theshape of an arrow.

In some embodiments, processing unit 110 may use an accuracy factor todetermine whether the pixel values correspond to an arrow. For example,processing unit 110 may determine whether the set of average pixelvalues match stored criteria (e.g., a set of pixel values, a storedimage, a pixel map, etc.) to within a degree of accuracy greater than athreshold value. In some embodiments, the accuracy factor may changedepending on a quality of the images. For example, a relatively lowaccuracy factor may be used if vehicle 200 is far away from a trafficlight and/or if only one of image capture devices 122, 124, 126 is ableto capture images of the traffic light. Similarly, a relatively highaccuracy factor may be used if vehicle 200 is close to the traffic lightand/or more than one (or all) of image capture devices 122, 124, 126 areable to capture images of the traffic light. In some embodiments, imageanalysis module 2830 may classify an object for one or more features.For example, image analysis module 2830 may classify an object as anarrow or round signal.

Through exemplary process 3000A, system 100 may be configured toidentify an arrow signal associated with a traffic light. The ability toidentify the arrow signal provides additional functionality to system100 that enhances the capabilities of system 100. In particular, sincean arrow signal may indicate different information than a non-arrowsignal (e.g., a plain round light) system 100 may be configured toprovide a particularized system response (e.g., acceptable to turn leftat this time, not acceptable to turn left at this time, etc. that mayotherwise be incorrect (e.g., green indicates go in all directions). Inanother example, system 100 may differentiate between two identifiedtraffic signal lights (e.g., a green round light and a red arrow light)to determine which traffic signal light should be followed. An exemplaryprocess for using the detection of an arrow signal to achieve thisfunctionality is described in more detail below.

FIG. 30B is a flowchart showing an exemplary process 3000B for causing asystem response based on the detection of an arrow signal in a trafficlight, consistent with disclosed embodiments. In an exemplaryembodiment, processing unit 110 may perform process 3000B to use aresult of process 3000A to control vehicle 200. For example, based on adetection of an arrow signal in a traffic light, processing unit 110 maycommand vehicle 200 to accelerate, turn, stop, change lanes, etc.

In step 3012, processing unit 110 may identify a traffic signal light.For example, processing unit 110 may identify a traffic signal light inone or more images captured by one or more image capture devices 122,124, 126. In some instances, more than one traffic signal light may beidentified in an image (e.g., a green round light and a red arrowlight).

At step 3014, processing unit 110 may determine a color of theidentified traffic signal light. Processing unit 110 may use a coloridentification process to analyze one or more of the plurality of imagesof the traffic light to determine and categorize a color of the trafficsignal light. For example, processing unit 110 may match image data tostored criteria to determine a color from a plurality of colorpossibilities (e.g., green, red, yellow, orange, etc.).

In one embodiment, processing unit 110 may identify a color of thepixels that include a pixel value indicating that the pixel is part ofthe shape of the traffic signal light. In another embodiment, one ormore image capture devices 122, 124, 126 may use clear pixels and red(or other color) pixels to assist with color differentiation. Forexample, in some embodiments, when a red light is present in an image,both the red and clear pixels may be used, whereas when a green (orother color) light is present, only the clear pixels may be used whilethe red pixels are ignored (e.g., defined as black, no color, not red,etc.). For example, when the light is red, the red pixels may match thecolor and be “used” to determine the color. As another example, when thelight is green, the clear pixels may be used to identify the colorand/or the red pixels are ignored to define the color as not red. Inthis way, processing unit 110 may differentiate between colors in imagesdepending on the type of pixels that are used to define a value of thelight within the pixel. Further, since the images may be aligned andaveraged with other images to achieve a higher resolution image, valuesfor ignored red pixels (or other pixels missing values) may be suppliedbased on information from other images.

It should be understood that the foregoing are examples and other coloridentification processes and/or techniques may be used. In someembodiments, processing unit 110 may be configured to determine one ormore other characteristics of the arrow and/or traffic signal. Forexample, processing unit 110 may determine whether the arrow is blinkingand/or steady. In another example, processing unit 110 may determinewhether the arrow has recently changed from another signal. For example,the plurality of images may include two identified signals, a previoussignal and a current signal (e.g., the identified arrow). These othercharacteristics may provide processing unit 110 with additionalinformation that may be helpful in determining how to interpret thearrow signal.

In step 3018, processing unit 110 may cause one or more system responsesbased on the color of the arrow (and/or other characteristic of thearrow and/or traffic signal). In one example, the system response maycorrespond to an indication of the arrow (e.g., green arrow—acceptableto turn, red arrow—not acceptable to turn, etc.). In another example,the system response may correspond to an indication that a particulartraffic signal does not apply because it is an arrow (e.g., the vehicleis traveling straight). The system response may be a navigationalresponse configured to change or continue an operating parameter ofvehicle 200. For example, processing unit 110 may provide a notice tothe driver of the vehicle (e.g., an audible sound or visual indicatorvia user interface 170), apply vehicle brakes, discontinue cruisecontrol, and/or initiate an automated turning maneuver. For example,processing unit 110 may provide control signals to one or more ofthrottling system 220, braking system 230, and steering system 240 tonavigate vehicle 200. In some embodiments, processing unit 110 maycontinue to analyze images to determine whether the system responseshould be interrupted (e.g., the arrow is no longer green—discontinuecruise control and apply brakes).

Through the exemplary disclosed processes 3000A and 3000B, system 100may have enhanced functionality in providing autonomous control ofvehicle 200. In particular, system 100 may be configured to react to atraffic signal based on detailed information provided by the trafficsignal and provide an appropriate system response that allows for safeoperation of vehicle 200. Further, while the above processes have beendescribed in the context of an arrow signal, it should be understoodthat one or more of the steps thereof may be used in similar processesto determine a system response to other objects and/or signals, such aspedestrian signals, no-turn-on-red signals, fixed road signs, streetsigns, traffic advisory boards, billboards, etc.

The foregoing description has been presented for purposes ofillustration. It is not exhaustive and is not limited to the preciseforms or embodiments disclosed. Modifications and adaptations will beapparent to those skilled in the art from consideration of thespecification and practice of the disclosed embodiments. Additionally,although aspects of the disclosed embodiments are described as beingstored in memory, one skilled in the art will appreciate that theseaspects can also be stored on other types of computer readable media,such as secondary storage devices, for example, hard disks or CD ROM, orother forms of RAM or ROM, USB media, DVD, Blu-ray, or other opticaldrive media.

Computer programs based on the written description and disclosed methodsare within the skill of an experienced developer. The various programsor program modules can be created using any of the techniques known toone skilled in the art or can be designed in connection with existingsoftware. For example, program sections or program modules can bedesigned in or by means of Net Framework, .Net Compact Framework (andrelated languages, such as Visual Basic, C, etc.), Java, C++,Objective-C, HTML, HTML/AJAX combinations, XML, or HTML with includedJava applets.

Moreover, while illustrative embodiments have been described herein, thescope of any and all embodiments having equivalent elements,modifications, omissions, combinations (e.g., of aspects across variousembodiments), adaptations and/or alterations as would be appreciated bythose skilled in the art based on the present disclosure. Thelimitations in the claims are to be interpreted broadly based on thelanguage employed in the claims and not limited to examples described inthe present specification or during the prosecution of the application.The examples are to be construed as non-exclusive. Furthermore, thesteps of the disclosed methods may be modified in any manner, includingby reordering steps and/or inserting or deleting steps. It is intended,therefore, that the specification and examples be considered asillustrative only, with a true scope and spirit being indicated by thefollowing claims and their full scope of equivalents.

What is claimed is:
 1. A traffic light detection system for a vehicle,the system comprising: at least one image capture device configured toacquire a plurality of images of an area forward of the vehicle, thearea including a traffic lamp fixture having at least one traffic light;a data interface; and at least one processing device configured to:receive the plurality of images via the data interface; align areas ofthe plurality of images corresponding to the traffic light, based on adetermined center of brightness; expand each pixel within the alignedareas into a subset of more than one pixel; determine a set of averagepixel values including an average pixel value for each set ofcorresponding expanded pixels within the aligned areas; and determine,based on the set of average pixel values, whether the traffic lightincludes an arrow.
 2. The traffic light detection system of claim 1,wherein expansion of each pixel within the aligned areas includesexpanding each pixel into a 3×3 matrix of pixels.
 3. The traffic lightdetection system of claim 1, wherein expansion of each pixel within thealigned areas includes expanding each pixel into a 4×4 matrix of pixels.4. The traffic light detection system of claim 1, wherein aligning areasof the plurality of images corresponding to the traffic light includesrotating the areas of the plurality of images.
 5. The traffic lightdetection system of claim 1, wherein the at least one processing deviceis further configured to: determine, when the traffic light includes anarrow, based on the set of average pixel values, a color of the arrow.6. The traffic light detection system of claim 5, wherein the at leastone processing device is further configured to: cause a system responsebased on the determination of the color of the arrow.
 7. The trafficlight detection system of claim 6, wherein the system response includesany of providing a notice to the driver of the vehicle, applying vehiclebrakes, discontinuing cruise control, and initiating an automatedturning maneuver.
 8. A vehicle, comprising: a body; at least one imagecapture device configured to acquire a plurality of images of an areaforward of the vehicle, the area including a traffic lamp fixture havingat least one traffic light; a data interface; and at least oneprocessing device configured to: receive the plurality of images via thedata interface; align areas of the plurality of images corresponding tothe traffic light, based on a determined center of brightness; expandeach pixel within the aligned areas into a subset of more than onepixel; determine a set of average pixel values including an averagepixel value for each set of corresponding expanded pixels within thealigned areas; and determine, based on the set of average pixel values,whether the traffic light includes an arrow.
 9. The vehicle of claim 8,wherein expansion of each pixel within the aligned areas includesexpanding each pixel into a 3×3 matrix of pixels.
 10. The vehicle ofclaim 8, wherein expansion of each pixel within the aligned areasincludes expanding each pixel into a 4×4 matrix of pixels.
 11. Thevehicle of claim 8, wherein aligning areas of the plurality of imagescorresponding to the traffic light includes rotating the areas of theplurality of images.
 12. The vehicle of claim 8, wherein the at leastone processing device is further configured to: determine, when thetraffic light includes an arrow, based on the set of average pixelvalues, a color of the arrow.
 13. A method for traffic light detection,the method comprising: acquiring, via at least one image capture device,a plurality of images of an area forward of a vehicle, the areaincluding a traffic lamp fixture having at least one traffic light;aligning areas of the plurality of images corresponding to the trafficlight, based on a determined center of brightness; expanding each pixelwithin the aligned areas into a subset of more than one pixel;determining a set of average pixel values including an average pixelvalue for each set of corresponding expanded pixels within the alignedareas; and determining, based on the set of average pixel values,whether the traffic light includes an arrow.
 14. The method of claim 13,wherein expansion of each pixel within the aligned areas includesexpanding each pixel into a 3×3 matrix of pixels.
 15. The method ofclaim 13, wherein expansion of each pixel within the aligned areasincludes expanding each pixel into a 4×4 matrix of pixels.
 16. Themethod of claim 13, wherein aligning areas of the plurality of imagescorresponding to the traffic light includes rotating the areas of theplurality of images.
 17. The method of claim 13, further comprising:determining, when the traffic light includes an arrow, based on the setof average pixel values, a color of the arrow.
 18. The method of claim17, further comprising: causing a system response based on thedetermination of the color of the arrow.
 19. The method of claim 18,wherein the system response includes any of providing a notice to thedriver of the vehicle, applying vehicle brakes, discontinuing cruisecontrol, and initiating an automated turning maneuver.